Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752132AbbKZC5o (ORCPT ); Wed, 25 Nov 2015 21:57:44 -0500 Received: from userp1040.oracle.com ([156.151.31.81]:49110 "EHLO userp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751699AbbKZC5m (ORCPT ); Wed, 25 Nov 2015 21:57:42 -0500 Date: Wed, 25 Nov 2015 21:57:24 -0500 From: Konrad Rzeszutek Wilk To: Bob Liu Cc: xen-devel@lists.xen.org, linux-kernel@vger.kernel.org, roger.pau@citrix.com, felipe.franciosi@citrix.com, axboe@fb.com, avanzini.arianna@gmail.com, rafal.mielniczuk@citrix.com, jonathan.davies@citrix.com, david.vrabel@citrix.com Subject: Re: [PATCH v5 00/10] xen-block: multi hardware-queues/rings support Message-ID: <20151126024320.GA3130@localhost.localdomain> References: <1447470739-18136-1-git-send-email-bob.liu@oracle.com> <20151125192507.GD19188@x230.dumpdata.com> <20151125205603.GB20267@x230.dumpdata.com> <20151125221234.GA22905@x230.dumpdata.com> <56566E3A.2010205@oracle.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <56566E3A.2010205@oracle.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-Source-IP: userv0021.oracle.com [156.151.31.71] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 7438 Lines: 151 On Thu, Nov 26, 2015 at 10:28:10AM +0800, Bob Liu wrote: > > On 11/26/2015 06:12 AM, Konrad Rzeszutek Wilk wrote: > > On Wed, Nov 25, 2015 at 03:56:03PM -0500, Konrad Rzeszutek Wilk wrote: > >> On Wed, Nov 25, 2015 at 02:25:07PM -0500, Konrad Rzeszutek Wilk wrote: > >>>> xen/blkback: separate ring information out of struct xen_blkif > >>>> xen/blkback: pseudo support for multi hardware queues/rings > >>>> xen/blkback: get the number of hardware queues/rings from blkfront > >>>> xen/blkback: make pool of persistent grants and free pages per-queue > >>> > >>> OK, got to those as well. I have put them in 'devel/for-jens-4.5' and > >>> are going to test them overnight before pushing them out. > >>> > >>> I see two bugs in the code that we MUST deal with: > >>> > >>> - print_stats () is going to show zero values. > >>> - the sysfs code (VBD_SHOW) aren't converted over to fetch data > >>> from all the rings. > >> > >> - kthread_run can't handle the two "name, i" arguments. I see: > >> > >> root 5101 2 0 20:47 ? 00:00:00 [blkback.3.xvda-] > >> root 5102 2 0 20:47 ? 00:00:00 [blkback.3.xvda-] > > > > And doing save/restore: > > > > xl save /tmp/A; > > xl restore /tmp/A; > > > > ends up us loosing the proper state and not getting the ring setup back. > > I see this is backend: > > > > [ 2719.448600] vbd vbd-22-51712: -1 guest requested 0 queues, exceeding the maximum of 3. > > > > And XenStore agrees: > > tool = "" > > xenstored = "" > > local = "" > > domain = "" > > 0 = "" > > domid = "0" > > name = "Domain-0" > > device-model = "" > > 0 = "" > > state = "running" > > error = "" > > backend = "" > > vbd = "" > > 2 = "" > > 51712 = "" > > error = "-1 guest requested 0 queues, exceeding the maximum of 3." > > > > .. which also leads to a memory leak as xen_blkbk_remove never gets > > called. > > I think which was already fix by your patch: > [PATCH RFC 2/2] xen/blkback: Free resources if connect_ring failed. Nope. I get that with or without the patch. I pushed the patches in git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git #devel/for-jens-4.5 tree. It also has some extra patches that should be soon going via the x86 tree. With the xen-blkback compiled with #define DEBUG 1 I see: [ 63.887741] xen-blkback: xen_blkbk_probe ffff880026a8cc00 1 [ 63.894302] xen-blkback: backend_changed ffff880026a8cc00 1 [ 63.895748] xen-blkback: frontend_changed ffff880026a8cc00 Initialising [ 63.922700] xen-blkback: xen_blkbk_probe ffff8800269da800 1 [ 63.927849] xen-blkback: backend_changed ffff8800269da800 1 [ 63.929117] xen-blkback: Successful creation of handle=ca00 (dom=1) [ 63.930605] xen-blkback: frontend_changed ffff8800269da800 Initialising [ 64.097161] xen-blkback: backend_changed ffff880026a8cc00 1 [ 64.098992] xen-blkback: Successful creation of handle=1600 (dom=1) [ 64.345913] device vif1.0 entered promiscuous mode [ 64.351469] IPv6: ADDRCONF(NETDEV_UP): vif1.0: link is not ready [ 64.538682] device vif1.0-emu entered promiscuous mode [ 64.546592] switch: port 3(vif1.0-emu) entered forwarding state [ 64.548357] switch: port 3(vif1.0-emu) entered forwarding state [ 79.544475] switch: port 3(vif1.0-emu) entered forwarding state [ 84.090637] switch: port 3(vif1.0-emu) entered disabled state [ 84.091545] device vif1.0-emu left promiscuous mode [ 84.092416] switch: port 3(vif1.0-emu) entered disabled state [ 89.286901] vif vif-1-0 vif1.0: Guest Rx ready [ 89.287921] IPv6: ADDRCONF(NETDEV_CHANGE): vif1.0: link becomes ready [ 89.288943] switch: port 2(vif1.0) entered forwarding state [ 89.289747] switch: port 2(vif1.0) entered forwarding state [ 89.456176] xen-blkback: frontend_changed ffff880026a8cc00 Closed [ 89.481945] xen-blkback: frontend_changed ffff8800269da800 Initialised [ 89.482802] xen-blkback: connect_ring /local/domain/1/device/vbd/51712 [ 89.484068] xen-blkback: backend/vbd/1/51712: using 2 queues, protocol 2 (x86_32-abi) persistent grants [ 89.532755] xen-blkback: connect /local/domain/1/device/vbd/51712 [ 89.541694] xen_update_blkif_status: name=[blkback.1.xvda-0] [ 89.542667] xen_update_blkif_status: name=[blkback.1.xvda-1] [ 89.561913] xen-blkback: frontend_changed ffff8800269da800 Connected .. so here the guest booted and now we are suspending it. [ 104.300579] switch: port 2(vif1.0) entered forwarding state [ 208.057752] xen-blkback: frontend_changed ffff880026a8cc00 Unknown [ 208.061282] xen-blkback: xen_blkbk_remove ffff880026a8cc00 1 [ 208.081888] xen-blkback: frontend_changed ffff8800269da800 Unknown [ 208.082759] xen-blkback: xen_blkbk_remove ffff8800269da800 1 [ 208.102745] switch: port 2(vif1.0) entered disabled state [ 208.109089] switch: port 2(vif1.0) entered disabled state [ 208.109934] device vif1.0 left promiscuous mode [ 208.110734] switch: port 2(vif1.0) entered disabled state We are done with suspend and we are resuming: [ 302.128132] xen-blkback: xen_blkbk_probe ffff880027fb1c00 2 [ 302.134915] xen-blkback: backend_changed ffff880027fb1c00 2 [ 302.136568] xen-blkback: frontend_changed ffff880027fb1c00 Initialising [ 302.165161] xen-blkback: xen_blkbk_probe ffff88001e2cac00 2 [ 302.170157] xen-blkback: backend_changed ffff88001e2cac00 2 [ 302.171420] xen-blkback: Successful creation of handle=ca00 (dom=2) [ 302.172873] xen-blkback: frontend_changed ffff88001e2cac00 Initialising [ 302.409007] xen-blkback: backend_changed ffff880027fb1c00 2 [ 302.411237] xen-blkback: Successful creation of handle=1600 (dom=2) [ 302.634468] device vif2.0 entered promiscuous mode [ 302.639871] IPv6: ADDRCONF(NETDEV_UP): vif2.0: link is not ready [ 302.822459] device vif2.0-emu entered promiscuous mode [ 302.830021] switch: port 3(vif2.0-emu) entered forwarding state [ 302.831568] switch: port 3(vif2.0-emu) entered forwarding state [ 302.849206] switch: port 3(vif2.0-emu) entered disabled state [ 302.850952] device vif2.0-emu left promiscuous mode [ 302.852567] switch: port 3(vif2.0-emu) entered disabled state [ 302.895719] xen-blkback: frontend_changed ffff88002e2cac00 Initialised [ 302.897237] xen-blkback: connect_ring /local/domain/2/device/vbd/51712 [ 302.899389] vbd vbd-2-51712: -1 guest requested 0 queues, exceeding the maximum of 3. [ 303.038086] xen-blkback: frontend_changed ffff88001e2cac00 Closing [ 303.073058] vif vif-2-0 vif2.0: Guest Rx ready [ 303.074057] IPv6: ADDRCONF(NETDEV_CHANGE): vif2.0: link becomes ready [ 303.075079] switch: port 2(vif2.0) entered forwarding state [ 303.075976] switch: port 2(vif2.0) entered forwarding state [ 306.204385] switch: port 2(vif2.0) entered disabled state [ 306.234820] xen-blkback: frontend_changed ffff880027fb1c00 Unknown [ 306.236307] xen-blkback: xen_blkbk_remove ffff880027fb1c00 2 [ 306.248684] xen-blkback: frontend_changed ffff88001e2cac00 Unknown [ 306.249958] xen-blkback: xen_blkbk_remove ffff88001e2cac00 2 [ 306.262379] switch: port 2(vif2.0) entered disabled state [ 306.263280] device vif2.0 left promiscuous mode [ 306.264210] switch: port 2(vif2.0) entered disabled state -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/