Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S965116AbbLOLTQ (ORCPT ); Tue, 15 Dec 2015 06:19:16 -0500 Received: from foss.arm.com ([217.140.101.70]:47991 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S964803AbbLOLTO (ORCPT ); Tue, 15 Dec 2015 06:19:14 -0500 Message-ID: <566FF730.8050609@arm.com> Date: Tue, 15 Dec 2015 11:19:12 +0000 From: Marc Zyngier Organization: ARM Ltd User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Icedove/31.7.0 MIME-Version: 1.0 To: Bhushan Bharat , "kvmarm@lists.cs.columbia.edu" , "kvm@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "linux-kernel@vger.kernel.org" Subject: Re: ARM64/KVM: Bad page state in process iperf References: <566FDEAD.6000707@arm.com> <566FE944.3090506@arm.com> In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 5275 Lines: 122 On 15/12/15 10:57, Bhushan Bharat wrote: > > >> -----Original Message----- >> From: Marc Zyngier [mailto:marc.zyngier@arm.com] >> Sent: Tuesday, December 15, 2015 3:50 PM >> To: Bhushan Bharat-R65777 ; >> kvmarm@lists.cs.columbia.edu; kvm@vger.kernel.org; linux-arm- >> kernel@lists.infradead.org; linux-kernel@vger.kernel.org >> Subject: Re: ARM64/KVM: Bad page state in process iperf >> >> On 15/12/15 09:53, Bhushan Bharat wrote: >>> Hi Mark, >>> >>>> -----Original Message----- >>>> From: Marc Zyngier [mailto:marc.zyngier@arm.com] >>>> Sent: Tuesday, December 15, 2015 3:05 PM >>>> To: Bhushan Bharat-R65777 ; >>>> kvmarm@lists.cs.columbia.edu; kvm@vger.kernel.org; linux-arm- >>>> kernel@lists.infradead.org; linux-kernel@vger.kernel.org >>>> Subject: Re: ARM64/KVM: Bad page state in process iperf >>>> >>>> On 15/12/15 03:46, Bhushan Bharat wrote: >>>>> >>>>> Hi All, >>>>> >>>>> I am running "iperf" in KVM guest on ARM64 machine and observing >>>>> below >>>> crash. >>>>> >>>>> ============================= >>>>> $iperf -c 3.3.3.3 -P 4 -t 0 -i 5 -w 90k >>>>> ------------------------------------------------------------ >>>>> Client connecting to 3.3.3.3, TCP port 5001 TCP window size: 180 >>>>> KByte (WARNING: requested 90.0 KByte) >>>>> ------------------------------------------------------------ >>>>> [ 3] local 3.3.3.1 port 51131 connected with 3.3.3.3 port 5001 [ >>>>> 6] local 3.3.3.1 port 51134 connected with 3.3.3.3 port 5001 [ 5] >>>>> local >>>>> 3.3.3.1 port 51133 connected with 3.3.3.3 port 5001 [ 4] local >>>>> 3.3.3.1 port 51132 connected with 3.3.3.3 port 5001 >>>>> [ 53.088567] random: nonblocking pool is initialized >>>>> [ ID] Interval Transfer Bandwidth >>>>> [ 3] 0.0- 5.0 sec 638 MBytes 1.07 Gbits/sec >>>>> [ 4] 35.0-40.0 sec 1.66 GBytes 2.85 Gbits/sec [ 5] 40.0-45.0 sec >>>>> 1.11 GBytes 1.90 Gbits/sec [ 4] 40.0-45.0 sec 1.16 GBytes 1.99 >>>>> Gbits/sec >>>>> [ 98.895207] BUG: Bad page state in process iperf pfn:0a584 >>>>> [ 98.896164] page:ffff780000296100 count:-1 mapcount:0 mapping: >>>> (null) index:0x0 >>>>> [ 98.897436] flags: 0x0() >>>>> [ 98.897885] page dumped because: nonzero _count >>>>> [ 98.898640] Modules linked in: >>>>> [ 98.899178] CPU: 0 PID: 1639 Comm: iperf Not tainted 4.1.8-00461- >>>> ge5431ad #141 >>>>> [ 98.900302] Hardware name: linux,dummy-virt (DT) >>>>> [ 98.901014] Call trace: >>>>> [ 98.901406] [] dump_backtrace+0x0/0x12c >>>>> [ 98.902522] [] show_stack+0x10/0x1c >>>>> [ 98.903441] [] dump_stack+0x8c/0xdc >>>>> [ 98.904202] [] bad_page+0xc4/0x114 >>>>> [ 98.904945] [] >> get_page_from_freelist+0x590/0x63c >>>>> [ 98.905871] [] >> __alloc_pages_nodemask+0xec/0x794 >>>>> [ 98.906791] [] skb_page_frag_refill+0x70/0xa8 >>>>> [ 98.907678] [] sk_page_frag_refill+0x20/0xd0 >>>>> [ 98.908550] [] tcp_sendmsg+0x1f8/0x9a8 >>>>> [ 98.909368] [] inet_sendmsg+0x5c/0xd0 >>>>> [ 98.910178] [] sock_sendmsg+0x14/0x58 >>>>> [ 98.911027] [] sock_write_iter+0x64/0xbc >>>>> [ 98.912119] [] __vfs_write+0xac/0x10c >>>>> [ 98.913126] [] vfs_write+0x90/0x1a0 >>>>> [ 98.913963] [] SyS_write+0x40/0xa0 >>>> >>>> This looks quite bad, but I don't see anything here that links it to >>>> KVM (apart from being a guest). Do you have any indication that this >>>> is due to KVM misbehaving? >>> >>> I never observed this issue in host Linux but observed this issue always in >> guest Linux. This issue does not comes immediately after I run "iperf" but >> after some time. >>> >>>> I'd appreciate a few more details. >>> >>> We have a networking hardware and we are directly assigning the h/w to >> guest. When using the same networking hardware in host it always works as >> expected (tried 100s of times). >>> Also this issue is not observed when we have only one vCPU in guest but >> seen when we have SMP guest. >> >> Can you reproduce the same issue without VFIO (using virtio, for example)? > > With virtio I have not observed this issue. > >> Is that platform VFIO? or PCI? > > It is not vfio-pci and vfio-platform. It is vfio-fls-mc (some > Freescale new hardware), similar to the lines of vfio-platform uses > same set of VFIO APIs used by vfio-pci/platform. Do you think this > can be some h/w specific issue. I have no idea, but by the look of it, something could be doing DMA on top of your guest page tables, which is not really expected. I suggest you carefully look at: 1) the DMA addresses that are passed to your device 2) the page tables that are programmed into the SMMU 3) the resulting translation Hopefully this will give you a clue about what is generating this. Thanks, M. -- Jazz is not dead. It just smells funny... -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/