Subject: Re: [PATCH 1/3] blk-mq: add an API to estimate hardware queue node
To: Christoph Hellwig <hch@infradead.org>
References: <68fed570910230ce847f8f3b685eeea399640a7f.1458941500.git.shli@fb.com>
 <20160329072443.GA18920@infradead.org>
 <20160329164722.GA1208161@devbig084.prn1.facebook.com>
 <56FAB243.5020804@fb.com> <20160329174437.GA451@infradead.org>
CC: Shaohua Li <shli@fb.com>, <linux-block@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <Kernel-team@fb.com>
From: Jens Axboe <axboe@fb.com>
Message-ID: <56FAEAD9.40908@fb.com>
Date: Tue, 29 Mar 2016 14:51:37 -0600
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.6.0
MIME-Version: 1.0
In-Reply-To: <20160329174437.GA451@infradead.org>
Content-Type: text/plain; charset="windows-1252"; format=flowed
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 1362
Lines: 29

On 03/29/2016 11:44 AM, Christoph Hellwig wrote:
> On Tue, Mar 29, 2016 at 10:50:11AM -0600, Jens Axboe wrote:
>>> This looks weird, shouldn't the cpu assignment be determined by block
>>> core (blk-mq) because block core decides how to use the queue?
>>
>> I agree, that belongs in the blk-mq proper, the driver should just follow
>> the rules outlined, not impose their own in this regard. It'll also help
>> with irq affinity mappings, once we get that in.
>
> It's not going to work that way unfortunately.  Lots of driver simply
> have no control over the underlying interrupts.  Think of any RDMA
> storage or other layer drivers - they get low level queues from a layer
> they don't control and need a block queue for each of them.
>
> My plan is to make the block layer follow what the networking layer
> does - get the low level queues / MSI-X pairs and then use the
> infrastructure in lib/cpu_rmap.c to figure out the number of queues
> and queue placement for them.

That sounds fine, as long as we get it in general infrastructure. Note 
that we need to be able to support sets of queues for blk-mq, once we 
start splitting them up (for scheduling). As long as we can map sets of 
queues to sets of CPUs, then that's not a concern.

For Shaohua's patches, not binding per-device data to the home node 
makes a lot of sense, though.

-- 
Jens Axboe