Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753136Ab3GAILa (ORCPT ); Mon, 1 Jul 2013 04:11:30 -0400 Received: from smtp.eu.citrix.com ([46.33.159.39]:1980 "EHLO SMTP.EU.CITRIX.COM" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753099Ab3GAIL1 (ORCPT ); Mon, 1 Jul 2013 04:11:27 -0400 X-IronPort-AV: E=Sophos;i="4.87,972,1363132800"; d="scan'208";a="6233852" Message-ID: <1372666283.14691.8.camel@zakaz.uk.xensource.com> Subject: Re: kernel panic in skb_copy_bits From: Ian Campbell To: Joe Jin CC: Alex Bligh , Eric Dumazet , Frank Blaschka , "David S. Miller" , , , , Xen Devel , Jan Beulich , "Stefano Stabellini" , Konrad Rzeszutek Wilk Date: Mon, 1 Jul 2013 09:11:23 +0100 In-Reply-To: <51D0F514.3070309@oracle.com> References: <51CBAA48.3080802@oracle.com> <1372311118.3301.214.camel@edumazet-glaptop> <51CD0E67.4000008@oracle.com> <6BFD5AF235F72F13CE646A0D@nimrod.local> <51D0F514.3070309@oracle.com> Organization: Citrix Systems, Inc. Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.4.4-1 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.30.203.1] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2642 Lines: 56 On Mon, 2013-07-01 at 11:18 +0800, Joe Jin wrote: > > A workaround is to turn off O_DIRECT use by Xen as that ensures > > the pages are copied. Xen 4.3 does this by default. > > > > I believe fixes for this are in 4.3 and 4.2.2 if using the > > qemu upstream DM. Note these aren't real fixes, just a workaround > > of a kernel bug. > > The guest is pvm, and disk model is xvbd, guest config file as below: Do you know which disk backend? The workaround Alex refers to went into qdisk but I think blkback could still suffer from a variant of the retransmit issue if you run it over iSCSI. > > To fix on a local build of xen you will need something like this: > > https://github.com/abligh/qemu-upstream-4.2-testing/commit/9a97c011e1a682eed9bc7195a25349eaf23ff3f9 > > and something like this (NB: obviously insert your own git > > repo and commit numbers) > > https://github.com/abligh/xen/commit/f5c344afac96ced8b980b9659fb3e81c4a0db5ca > > > > I think this only for pvhvm/hvm? No, the underlying issue affects any PV device which is run over a network protocol (NFS, iSCSI etc). In effect a delayed retransmit can cross over the deayed ack and cause I/O to be completed while retransmits are pending, such as is described in http://www.spinics.net/lists/linux-nfs/msg34913.html (the original NFS variant). The problem is that because Xen PV drivers often unmap the page on I/O completion you get a crash (page fault) on the retransmit. The issue also affects native but in that case the symptom is "just" a corrupt packet on the wire. I tried to address this with my "skb destructor" series but unfortunately I got bogged down on the details, then I had to take time out to look into some other stuff and never managed to get back into it. I'd be very grateful if there was someone who could pick up that work (Alex gave some useful references in another reply to this thread) Some PV disk backends (e.g. blktap2) have worked around this by using grant copy instead of grant map, others (e.g. qdisk) have disabled O_DIRECT so that the pages are copied into the dom0 page cache and transmitted from there. We were discussing recently the possibility of mapping all ballooned out pages to a single read-only scratch page instead of leaving them empty in the page tables, this would cause the Xen case to revert to the native case. I think Thanos was going to take a look into this. Ian. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/