Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S1752750AbbKZHJT (ORCPT <rfc822;w@1wt.eu>);
	Thu, 26 Nov 2015 02:09:19 -0500
Received: from aserp1040.oracle.com ([141.146.126.69]:34229 "EHLO
	aserp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1752253AbbKZHJO (ORCPT
	<rfc822;linux-kernel@vger.kernel.org>);
	Thu, 26 Nov 2015 02:09:14 -0500
Message-ID: <5656B00E.6070303@oracle.com>
Date: Thu, 26 Nov 2015 15:09:02 +0800
From: Bob Liu <bob.liu@oracle.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130308 Thunderbird/17.0.4
MIME-Version: 1.0
To: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
CC: xen-devel@lists.xen.org, linux-kernel@vger.kernel.org,
        roger.pau@citrix.com, felipe.franciosi@citrix.com, axboe@fb.com,
        avanzini.arianna@gmail.com, rafal.mielniczuk@citrix.com,
        jonathan.davies@citrix.com, david.vrabel@citrix.com
Subject: Re: [PATCH v5 00/10] xen-block: multi hardware-queues/rings support
References: <1447470739-18136-1-git-send-email-bob.liu@oracle.com> <20151125192507.GD19188@x230.dumpdata.com> <20151125205603.GB20267@x230.dumpdata.com> <20151125221234.GA22905@x230.dumpdata.com> <56566E3A.2010205@oracle.com> <20151126024320.GA3130@localhost.localdomain>
In-Reply-To: <20151126024320.GA3130@localhost.localdomain>
Content-Type: multipart/mixed;
 boundary="------------020000070800000003070609"
X-Source-IP: userv0021.oracle.com [156.151.31.71]
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 6104
Lines: 191

This is a multi-part message in MIME format.
--------------020000070800000003070609
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit


On 11/26/2015 10:57 AM, Konrad Rzeszutek Wilk wrote:
> On Thu, Nov 26, 2015 at 10:28:10AM +0800, Bob Liu wrote:
>>
>> On 11/26/2015 06:12 AM, Konrad Rzeszutek Wilk wrote:
>>> On Wed, Nov 25, 2015 at 03:56:03PM -0500, Konrad Rzeszutek Wilk wrote:
>>>> On Wed, Nov 25, 2015 at 02:25:07PM -0500, Konrad Rzeszutek Wilk wrote:
>>>>>>   xen/blkback: separate ring information out of struct xen_blkif
>>>>>>   xen/blkback: pseudo support for multi hardware queues/rings
>>>>>>   xen/blkback: get the number of hardware queues/rings from blkfront
>>>>>>   xen/blkback: make pool of persistent grants and free pages per-queue
>>>>>
>>>>> OK, got to those as well. I have put them in 'devel/for-jens-4.5' and
>>>>> are going to test them overnight before pushing them out.
>>>>>
>>>>> I see two bugs in the code that we MUST deal with:
>>>>>
>>>>>  - print_stats () is going to show zero values.
>>>>>  - the sysfs code (VBD_SHOW) aren't converted over to fetch data
>>>>>    from all the rings.
>>>>
>>>> - kthread_run can't handle the two "name, i" arguments. I see:
>>>>
>>>> root      5101     2  0 20:47 ?        00:00:00 [blkback.3.xvda-]
>>>> root      5102     2  0 20:47 ?        00:00:00 [blkback.3.xvda-]
>>>
>>> And doing save/restore:
>>>
>>> xl save <id> /tmp/A;
>>> xl restore /tmp/A;
>>>
>>> ends up us loosing the proper state and not getting the ring setup back.
>>> I see this is backend:
>>>
>>> [ 2719.448600] vbd vbd-22-51712: -1 guest requested 0 queues, exceeding the maximum of 3.
>>>
>>> And XenStore agrees:
>>> tool = ""
>>>  xenstored = ""
>>> local = ""
>>>  domain = ""
>>>   0 = ""
>>>    domid = "0"
>>>    name = "Domain-0"
>>>    device-model = ""
>>>     0 = ""
>>>      state = "running"
>>>    error = ""
>>>     backend = ""
>>>      vbd = ""
>>>       2 = ""
>>>        51712 = ""
>>>         error = "-1 guest requested 0 queues, exceeding the maximum of 3."
>>>
>>> .. which also leads to a memory leak as xen_blkbk_remove never gets
>>> called.
>>
>> I think which was already fix by your patch:
>> [PATCH RFC 2/2] xen/blkback: Free resources if connect_ring failed.
> 
> Nope. I get that with or without the patch.
> 

Attached patch should fix this issue. 

-- 
Regards,
-Bob

--------------020000070800000003070609
Content-Type: text/x-patch;
 name="0001-xen-blkfront-fix-compile-error.patch"
Content-Transfer-Encoding: 8bit
Content-Disposition: attachment;
 filename="0001-xen-blkfront-fix-compile-error.patch"

>From f297a05fc27fb0bc9a3ed15407f8cc6ffd5e2a00 Mon Sep 17 00:00:00 2001
From: Bob Liu <bob.liu@oracle.com>
Date: Wed, 25 Nov 2015 14:56:32 -0500
Subject: [PATCH 1/2] xen:blkfront: fix compile error
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Fix this build error:
drivers/block/xen-blkfront.c: In function ‘blkif_free’:
drivers/block/xen-blkfront.c:1234:6: error: ‘struct blkfront_info’ has no
member named ‘ring’ info->ring = NULL;

Signed-off-by: Bob Liu <bob.liu@oracle.com>
---
 drivers/block/xen-blkfront.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c
index 625604d..ef5ce43 100644
--- a/drivers/block/xen-blkfront.c
+++ b/drivers/block/xen-blkfront.c
@@ -1231,7 +1231,7 @@ static void blkif_free(struct blkfront_info *info, int suspend)
 		blkif_free_ring(&info->rinfo[i]);
 
 	kfree(info->rinfo);
-	info->ring = NULL;
+	info->rinfo = NULL;
 	info->nr_rings = 0;
 }
 
-- 
1.8.3.1


--------------020000070800000003070609
Content-Type: text/x-patch;
 name="0002-xen-blkfront-realloc-ring-info-in-blkif_resume.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="0002-xen-blkfront-realloc-ring-info-in-blkif_resume.patch"

>From aab0bb1690213e665966ea22b021e0eeaacfc717 Mon Sep 17 00:00:00 2001
From: Bob Liu <bob.liu@oracle.com>
Date: Wed, 25 Nov 2015 17:52:55 -0500
Subject: [PATCH 2/2] xen/blkfront: realloc ring info in blkif_resume

Need to reallocate ring info in the resume path, because info->rinfo was freed
in blkif_free(). And 'multi-queue-max-queues' backend reports may have been
changed.

Signed-off-by: Bob Liu <bob.liu@oracle.com>
---
 drivers/block/xen-blkfront.c | 28 +++++++++++++++++++++++++++-
 1 file changed, 27 insertions(+), 1 deletion(-)

diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c
index ef5ce43..9634a65 100644
--- a/drivers/block/xen-blkfront.c
+++ b/drivers/block/xen-blkfront.c
@@ -1926,12 +1926,38 @@ static int blkif_recover(struct blkfront_info *info)
 static int blkfront_resume(struct xenbus_device *dev)
 {
 	struct blkfront_info *info = dev_get_drvdata(&dev->dev);
-	int err;
+	int err = 0;
+	unsigned int max_queues = 0, r_index;
 
 	dev_dbg(&dev->dev, "blkfront_resume: %s\n", dev->nodename);
 
 	blkif_free(info, info->connected == BLKIF_STATE_CONNECTED);
 
+	err = xenbus_gather(XBT_NIL, info->xbdev->otherend,
+			"multi-queue-max-queues", "%u", &max_queues, NULL);
+	if (err)
+		max_queues = 1;
+
+	info->nr_rings = min(max_queues, xen_blkif_max_queues);
+	/* We need at least one ring. */
+	if (!info->nr_rings)
+		info->nr_rings = 1;
+
+	info->rinfo = kzalloc(sizeof(struct blkfront_ring_info) * info->nr_rings, GFP_KERNEL);
+	if (!info->rinfo)
+		return -ENOMEM;
+
+	for (r_index = 0; r_index < info->nr_rings; r_index++) {
+		struct blkfront_ring_info *rinfo;
+
+		rinfo = &info->rinfo[r_index];
+		INIT_LIST_HEAD(&rinfo->indirect_pages);
+		INIT_LIST_HEAD(&rinfo->grants);
+		rinfo->dev_info = info;
+		INIT_WORK(&rinfo->work, blkif_restart_queue);
+		spin_lock_init(&rinfo->ring_lock);
+	}
+
 	err = talk_to_blkback(dev, info);
 
 	/*
-- 
1.8.3.1


--------------020000070800000003070609--
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/