Received: by 2002:a25:4158:0:0:0:0:0 with SMTP id o85csp4480649yba; Sun, 19 May 2019 20:48:42 -0700 (PDT) X-Google-Smtp-Source: APXvYqy6u/9ZzZMo/f7dRPh/mOhgjoePwmJU6hHW62/k5tJMHtCxVtLD/ByCiYOBaH27gFcz4LB4 X-Received: by 2002:a17:902:5a47:: with SMTP id f7mr48273843plm.321.1558324122859; Sun, 19 May 2019 20:48:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558324122; cv=none; d=google.com; s=arc-20160816; b=WqA34c/kJJvc6uEKplq3IV2HUa1l9aNi+sAkjM3V179lrsTzTW3se5MfMQHheWPnoZ hZkMtjFOAWGyV+eW69ztGVobxZjV5a5gXr+psfPI4AfyToEMT1Xb78Ccd46jtuKlXphq YzkyQA3y4MJVPYmBHoCgb6+T0atAd0qHNixexDTmEwfjIOrjLJY+C51cYvV+s9RLsaGA Tc8Csvu9ccNmEV8T7wYM9IRoqEo0KEPbXpAXVztPRm1svZJZnQKVquUJKbeYBqytQSMI yQeX1jzAaZxHS+liOMku1SfbfaANhKxXhkOko93gKngGXClVfsDdjXUSPEeZJ7wO1ZOC dp+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :message-id:date:references:in-reply-to:subject:cc:to:from; bh=W5Q2Y/dYsRU4FNEffnab7RKvy1W12a6T+VQ5QPe06Ak=; b=N+tjy8DU8n5cI+k/gjKpQk+9+yGA3bbyiaOvi7cyy+MMn1wE5pW9cUum6aqtoFfBC4 yDv8tbt2gbyUgOfTINBczWMLSmF19B7JV8wl2A/VBwyjqmYdswqaWxJqBjfsgU/4m4kO KxXeDxLKNhMsfsZtKjVfPJJ5V++9f7Es7EzNj0lARhG9tK5MaZsTZJJSIswwtaR7U4vY 68FzAgWRXpsculrGMdENdjtqCOADHHn652f4rSFELyQZ0eUv708sj/AxX5Rie8xl5N++ zVMaTqBXuw2NNyad0MTcPrb8RhOE4CfQlHX4PdHYs7ywc8KeLEO2vHurWM8rs/r5rlqg /tew== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id z18si16074446plo.72.2019.05.19.20.48.28; Sun, 19 May 2019 20:48:42 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727576AbfETCC2 convert rfc822-to-8bit (ORCPT + 99 others); Sun, 19 May 2019 22:02:28 -0400 Received: from ozlabs.org ([203.11.71.1]:60021 "EHLO ozlabs.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727419AbfETCC1 (ORCPT ); Sun, 19 May 2019 22:02:27 -0400 Received: from authenticated.ozlabs.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.ozlabs.org (Postfix) with ESMTPSA id 456hwv6bSbz9sB8; Mon, 20 May 2019 12:02:23 +1000 (AEST) From: Michael Ellerman To: bharata@linux.ibm.com, srikanth Cc: linuxppc-dev@lists.ozlabs.org, bharata@linux.vnet.ibm.com, linux-kernel@vger.kernel.org, linux-next@vger.kernel.org, npiggin@gmail.com, aneesh.kumar@linux.ibm.com Subject: Re: PROBLEM: Power9: kernel oops on memory hotunplug from ppc64le guest In-Reply-To: <20190518141434.GA22939@in.ibm.com> References: <16a7a635-c592-27e2-75b4-d02071833278@linux.vnet.ibm.com> <20190518141434.GA22939@in.ibm.com> Date: Mon, 20 May 2019 12:02:23 +1000 Message-ID: <878sv1993k.fsf@concordia.ellerman.id.au> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8BIT Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Bharata B Rao writes: > On Thu, May 16, 2019 at 07:44:20PM +0530, srikanth wrote: >> Hello, >> >> On power9 host, performing memory hotunplug from ppc64le guest results in >> kernel oops. >> >> Kernel used : https://github.com/torvalds/linux/tree/v5.1 built using >> ppc64le_defconfig for host and ppc64le_guest_defconfig for guest. >> >> Recreation steps: >> >> 1. Boot a guest with below mem configuration: >>   33554432 >>   8388608 >>   4194304 >>   >>     >>       >>     >>   >> >> 2. From host hotplug 8G memory -> verify memory hotadded succesfully -> now >> reboot guest -> once guest comes back try to unplug 8G memory >> >> mem.xml used: >> >> >> 8 >> 0 >> >> >> >> Memory attach and detach commands used: >>     virsh attach-device vm1 ./mem.xml --live >>     virsh detach-device vm1 ./mem.xml --live >> >> Trace seen inside guest after unplug, guest just hangs there forever: >> >> [   21.962986] kernel BUG at arch/powerpc/mm/pgtable-frag.c:113! >> [   21.963064] Oops: Exception in kernel mode, sig: 5 [#1] >> [   21.963090] LE PAGE_SIZE=64K MMU=Radix MMU=Hash SMP NR_CPUS=2048 NUMA >> pSeries >> [   21.963131] Modules linked in: xt_tcpudp iptable_filter squashfs fuse >> vmx_crypto ib_iser rdma_cm iw_cm ib_cm ib_core libiscsi scsi_transport_iscsi >> ip_tables x_tables autofs4 btrfs zstd_decompress zstd_compress lzo_compress >> raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx >> xor raid6_pq multipath crc32c_vpmsum >> [   21.963281] CPU: 11 PID: 316 Comm: kworker/u64:5 Kdump: loaded Not >> tainted 5.1.0-dirty #2 >> [   21.963323] Workqueue: pseries hotplug workque pseries_hp_work_fn >> [   21.963355] NIP:  c000000000079e18 LR: c000000000c79308 CTR: >> 0000000000008000 >> [   21.963392] REGS: c0000003f88034f0 TRAP: 0700   Not tainted (5.1.0-dirty) >> [   21.963422] MSR:  800000000282b033   CR: >> 28002884  XER: 20040000 >> [   21.963470] CFAR: c000000000c79304 IRQMASK: 0 >> [   21.963470] GPR00: c000000000c79308 c0000003f8803780 c000000001521000 >> 0000000000fff8c0 >> [   21.963470] GPR04: 0000000000000001 00000000ffe30005 0000000000000005 >> 0000000000000020 >> [   21.963470] GPR08: 0000000000000000 0000000000000001 c00a000000fff8e0 >> c0000000016d21a0 >> [   21.963470] GPR12: c0000000016e7b90 c000000007ff2700 c00a000000a00000 >> c0000003ffe30100 >> [   21.963470] GPR16: c0000003ffe30000 c0000000014aa4de c00a0000009f0000 >> c0000000016d21b0 >> [   21.963470] GPR20: c0000000014de588 0000000000000001 c0000000016d21b8 >> c00a000000a00000 >> [   21.963470] GPR24: 0000000000000000 ffffffffffffffff c00a000000a00000 >> c0000003ffe96000 >> [   21.963470] GPR28: c00a000000a00000 c00a000000a00000 c0000003fffec000 >> c00a000000fff8c0 >> [   21.963802] NIP [c000000000079e18] pte_fragment_free+0x48/0xd0 >> [   21.963838] LR [c000000000c79308] remove_pagetable+0x49c/0x5b4 >> [   21.963873] Call Trace: >> [   21.963890] [c0000003f8803780] [c0000003ffe997f0] 0xc0000003ffe997f0 >> (unreliable) >> [   21.963933] [c0000003f88037b0] [0000000000000000] (null) >> [   21.963969] [c0000003f88038c0] [c00000000006f038] >> vmemmap_free+0x218/0x2e0 >> [   21.964006] [c0000003f8803940] [c00000000036f100] >> sparse_remove_one_section+0xd0/0x138 >> [   21.964050] [c0000003f8803980] [c000000000383a50] >> __remove_pages+0x410/0x560 >> [   21.964093] [c0000003f8803a90] [c000000000c784d8] >> arch_remove_memory+0x68/0xdc >> [   21.964136] [c0000003f8803ad0] [c000000000385d74] >> __remove_memory+0xc4/0x110 >> [   21.964180] [c0000003f8803b10] [c0000000000d44e4] >> dlpar_remove_lmb+0x94/0x140 >> [   21.964223] [c0000003f8803b50] [c0000000000d52b4] >> dlpar_memory+0x464/0xd00 >> [   21.964259] [c0000003f8803be0] [c0000000000cd5c0] >> handle_dlpar_errorlog+0xc0/0x190 >> [   21.964303] [c0000003f8803c50] [c0000000000cd6bc] >> pseries_hp_work_fn+0x2c/0x60 >> [   21.964346] [c0000003f8803c80] [c00000000013a4a0] >> process_one_work+0x2b0/0x5a0 >> [   21.964388] [c0000003f8803d10] [c00000000013a818] >> worker_thread+0x88/0x610 >> [   21.964434] [c0000003f8803db0] [c000000000143884] kthread+0x1a4/0x1b0 >> [   21.964468] [c0000003f8803e20] [c00000000000bdc4] >> ret_from_kernel_thread+0x5c/0x78 >> [   21.964506] Instruction dump: >> [   21.964527] fbe1fff8 f821ffd1 78638502 78633664 ebe90000 7fff1a14 >> 395f0020 813f0020 >> [   21.964569] 7d2907b4 7d2900d0 79290fe0 69290001 <0b090000> 7c0004ac >> 7d205028 3129ffff >> [   21.964613] ---[ end trace aaa571aa1636fee6 ]--- >> [   21.966349] >> [   21.966383] Sending IPI to other CPUs >> [   21.978335] IPI complete >> [   21.981354] kexec: Starting switchover sequence. >> I'm in purgatory > > git bisect points to > > commit 4231aba000f5a4583dd9f67057aadb68c3eca99d > Author: Nicholas Piggin > Date: Fri Jul 27 21:48:17 2018 +1000 > > powerpc/64s: Fix page table fragment refcount race vs speculative references > > The page table fragment allocator uses the main page refcount racily > with respect to speculative references. A customer observed a BUG due > to page table page refcount underflow in the fragment allocator. This > can be caused by the fragment allocator set_page_count stomping on a > speculative reference, and then the speculative failure handler > decrements the new reference, and the underflow eventually pops when > the page tables are freed. > > Fix this by using a dedicated field in the struct page for the page > table fragment allocator. > > Fixes: 5c1f6ee9a31c ("powerpc: Reduce PTE table memory wastage") > Cc: stable@vger.kernel.org # v3.10+ That's the commit that added the BUG_ON(), so prior to that you won't see the crash. cheers