From: "Stephen  Bates" <sbates@raithlin.com>
To: Jens Axboe <axboe@kernel.dk>,
        "linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
CC: "osandov@fb.com" <osandov@fb.com>,
        "damien.lemoal@wdc.com" <damien.lemoal@wdc.com>
Subject: Re: [PATCH] blk-mq: Improvements to the hybrid polling sleep time
 calculation
Thread-Topic: [PATCH] blk-mq: Improvements to the hybrid polling sleep time
 calculation
Thread-Index: AQHTGorMhi9dtMloh0ixIeJiETs4MqKQ08KAgApK1AA=
Date: Tue, 29 Aug 2017 15:33:15 +0000
Message-ID: <B1195188-245E-42BE-AC5D-2070F33E1D93@raithlin.com>
References: <1503326134-3862-1-git-send-email-sbates@raithlin.com>
 <54dad77e-18d7-eb64-35fb-670fecc83ce7@kernel.dk>
In-Reply-To: <54dad77e-18d7-eb64-35fb-670fecc83ce7@kernel.dk>
Accept-Language: en-CA, en-US
Content-Language: en-US
user-agent: Microsoft-MacOutlook/f.25.0.170815
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="utf-8"
Content-ID: <C9BBB208D0521B4D8428C4FDED165311@CANPRD01.PROD.OUTLOOK.COM>
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Aug 2017 15:33:15.3774
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 18519031-7ff4-4cbb-bbcb-c3252d330f4b
X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR01MB0682
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by nfs id v7TGNeJ8027474
Content-Length: 2825
Lines: 61

>> From: Stephen Bates <sbates@raithlin.com>
>> 
>> Hybrid polling currently uses half the average completion time as an
>> estimate of how long to poll for. We can improve upon this by noting
>> that polling before the minimum completion time makes no sense. Add a
>> sysfs entry to use this fact to improve CPU utilization in certain
>> cases.
>> 
>> At the same time the minimum is a bit too long to sleep for since we
>> must factor in OS wake time for the thread. For now allow the user to
>> set this via a second sysfs entry (in nanoseconds).
>> 
>> Testing this patch on Intel Optane SSDs showed that using the minimum
>> rather than half reduced CPU utilization from 59% to 38%. Tuning
>> this via the wake time adjustment allowed us to trade CPU load for
>> latency. For example
>> 
>> io_poll	 delay	hyb_use_min adjust	latency	CPU load
>> 1	 -1	N/A	    N/A		8.4	100%
>> 1	 0	0	    N/A		8.4	57%
>> 1	 0	1	    0		10.3	34%
>> 1	 9	1	    1000	9.9	37%
>> 1	 0	1	    2000	8.4	47%
>> 1	 0	1	    10000	8.4	100%
>> 
>> Ideally we will extend this to auto-calculate the wake time rather
>> than have it set by the user.
>
> I don't like this, it's another weird knob that will exist but that
> no one will know how to use. For most of the testing I've done
> recently, hybrid is a win over busy polling - hence I think we should
> make that the default. 60% of mean has also, in testing, been shown
> to be a win. So that's an easy fix/change we can consider.

I do agree that the this is a hard knob to tune. I am however not happy that the current hybrid default may mean we are polling well before the minimum completion time. That just seems like a waste of CPU resources to me. I do agree that turning on hybrid as the default and perhaps bumping up the default is a good idea.

> To go beyond that, I'd much rather see us tracking the time waste.
> If we consider the total completion time of an IO to be A+B+C, where:
>
> A	Time needed to go to sleep
> B	Sleep time
> C	Time needed to wake up
>
> then we could feasibly track A+C. We already know how long the IO
> will take to complete, as we track that. At that point we'd have
> a full picture of how long we should sleep.

Yes, this is where I was thinking of taking this functionality in the long term. It seems like tracking C is something other parts of the kernel might need. Does anyone know of any existing code in this space?

> Bonus points for informing the lower level scheduler of this as
> well. If the CPU is going idle, we'll enter some sort of power
> state in the processor. If we were able to pass in how long we
> expect to sleep, we could be making better decisions here.

Yup. Again, this seems like something more general that just the block-layer. I will do some digging and see/if anything is available to leverage here.

Cheers
Stephen