From: KY Srinivasan <kys@microsoft.com>
To: Vitaly Kuznetsov <vkuznets@redhat.com>, Dexuan Cui <decui@microsoft.com>
CC: Haiyang Zhang <haiyangz@microsoft.com>,
        "devel@linuxdriverproject.org" <devel@linuxdriverproject.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH 5/6] Drivers: hv: vmbus: distribute subchannels among all
 vcpus
Thread-Topic: [PATCH 5/6] Drivers: hv: vmbus: distribute subchannels among all
 vcpus
Thread-Index: AQHQfm3Kd3+YMWARzUmM10zJuDODyZ1cXT8w
Date: Fri, 24 Apr 2015 16:46:52 +0000
Message-ID: <BN1PR0301MB0707CF5E57279A9E0244684AA0EC0@BN1PR0301MB0707.namprd03.prod.outlook.com>
References: <1429626460-7947-1-git-send-email-vkuznets@redhat.com>
	<1429626460-7947-6-git-send-email-vkuznets@redhat.com>
	<091d0fd6321f4dd490e61a574d5b5b50@SIXPR30MB031.064d.mgd.msft.net>
 <87oamdgalr.fsf@vitty.brq.redhat.com>
In-Reply-To: <87oamdgalr.fsf@vitty.brq.redhat.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 8BIT
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Apr 2015 16:46:52.2829
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN1PR0301MB0769
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 3393
Lines: 81


> -----Original Message-----
> From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com]
> Sent: Friday, April 24, 2015 2:05 AM
> To: Dexuan Cui
> Cc: KY Srinivasan; Haiyang Zhang; devel@linuxdriverproject.org; linux-
> kernel@vger.kernel.org
> Subject: Re: [PATCH 5/6] Drivers: hv: vmbus: distribute subchannels among
> all vcpus
> 
> Dexuan Cui <decui@microsoft.com> writes:
> 
> >> -----Original Message-----
> >> From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com]
> >> Sent: Tuesday, April 21, 2015 22:28
> >> To: KY Srinivasan
> >> Cc: Haiyang Zhang; devel@linuxdriverproject.org; linux-
> >> kernel@vger.kernel.org; Dexuan Cui
> >> Subject: [PATCH 5/6] Drivers: hv: vmbus: distribute subchannels among all
> >> vcpus
> >>
> >> Primary channels are distributed evenly across all vcpus we have. When
> the
> >> host asks us to create subchannels it usually makes us num_cpus-1 offers
> >
> > Hi Vitaly,
> > AFAIK, in the VSP of storvsc, the number of subchannel is
> >  (the_number_of_vcpus - 1) / 4.
> >
> > This means for a 8-vCPU guest, there is only 1 subchannel.
> >
> > Your new algorithm tends to make the vCPUs with small-number busier:
> > e.g., in the 8-vCPU case, assuming we have 4 SCSI controllers:
> > vCPU0: scsi0's PrimaryChannel (P)
> > vCPU1: scsi0's SubChannel (S) + scsi1's P
> > vCPU2: scsi1's S + scsi2's P
> > vCPU3: scsi2's S + scsi3's P
> > vCPU4: scsi3's S
> > vCPU5, 6 and 7 are idle.
> >
> > In this special case, the existing algorithm is better. :-)
> >
> > However, I do like this idea in your patch, that is, making sure a device's
> > primary/sub channels are assigned to differents vCPUs.
> 
> Under special circumstances with the current code we can end up with
> having all subchannels on the same vCPU with the primary channel I guess
> :-) This is not something common, but possible.
> 
> >
> > I'm just wondering if we should use an even better (and complex)
> > algorithm :-)
> 
> The question here is - does sticking to the current vCPU help? If it
> does, I can suggest the following (I think I even mentioned that in my
> PATCH 00): first we try to find a (sub)channel with target_cpu ==
> current_vcpu and only when we fail we do the round robin. I'd like to
> hear K.Y.'s opinion here as he's the original author :-)

Sorry for the delayed response. Initially I had implemented a scheme that would 
pick an outgoing CPU that was closest to the CPU on which the request came (to maintain
cache locality especially on NUMA systems). I changed this algorithm to spread the load
more uniformly as we were trying to improve Linux IOPS on Azure XIO
(premium storage). We are currently testing
this code on our Converged Offering - CPS and I am finding that the perf as measured by IOS has regressed.
I have not narrowed the reason for this regression and it may very well be the change in the 
algorithm for selecting the outgoing channel. In general, I don't think the logic here needs to be 
exact and locality (being on the same CPU or within the same NUMA node) is important. Any change
to this algorithm will have to be validated on different MSFT environments (Azure XIO, CPS etc.).

Regards,

K. Y

 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/