Message-ID: <55A67F11.1030709@sandisk.com>
Date: Wed, 15 Jul 2015 08:41:05 -0700
From: Bart Van Assche <bart.vanassche@sandisk.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0
MIME-Version: 1.0
To: Thomas Gleixner <tglx@linutronix.de>,
        Christoph Hellwig <hch@infradead.org>
CC: <ksummit-discuss@lists.linuxfoundation.org>, <linux-rdma@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <linux-nvme@lists.infradead.org>,
        Jens Axboe <axboe@kernel.dk>
Subject: Re: [Ksummit-discuss] [TECH TOPIC] IRQ affinity
References: <20150715120708.GA24534@infradead.org> <alpine.DEB.2.11.1507151408570.18576@nanos>
In-Reply-To: <alpine.DEB.2.11.1507151408570.18576@nanos>
Content-Type: text/plain; charset="windows-1252"; format=flowed
Content-Transfer-Encoding: 7bit
SN1PR02MB1373: X-MS-Exchange-Organization-RulesExecuted
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jul 2015 15:41:06.9923
 (UTC)
X-MS-Exchange-CrossTenant-Id: fcd9ea9c-ae8c-460c-ab3c-3db42d7ac64d
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=fcd9ea9c-ae8c-460c-ab3c-3db42d7ac64d;Ip=[63.163.107.173];Helo=[milsmgep12.sandisk.com]
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1PR02MB1373
X-Microsoft-Exchange-Diagnostics: 1;SN1PR02MB1695;2:ykzY+DGxG9Zed1L/igixpBXUmmvD28VqQK01awulTQRFTkhMio2oad/j7ytilByq;3:nvf7Nj16LhVZgwvFFUI3KJdPZA5V0RF/jpBnv/8rir2VQt2eQjpxDPWwb4gSgN9n34HZNLKIlFr8q0jCAFjialQJlvAlw38q/1jLJ61bKemRPTEVZCQNgjwqvB+iLffN+dHaUrlBIO0lhRB2gE7SnNpLbrCkeqjSS1AOiMg04P9VlnKgpCwIt0P7mkLMPge6CEkrrHeYUWcdZUbleB6peELt/Bagb8q7LvCRS5lsrefKyuQE2bRJZ3wERFFAkAGQ;25:fV+UqH1LPruk0BnMLXGu4ppktOgxSgYK+QBTozfXWyCw2g4VwmWUmbfMWfXT1ieQzuHCCKFXV2zdD3Br+KWcUpIX/xoD0lx3tzG0VYIH5c1JaNZg/A37bS5r/FbmkwdadLD6qYVyNR/+mcXmO8/P37UjNYHfdtCvioZZLqBp+qUlUU4wcJEik4vOu22OETlqxrWQ1WIoXzMU4Jwu27hLd5843qRWzQHrM3YrlcfCZoRsZjH4P4ABY1dl39RymiL8hCCeyxwiDwA4230uqFhr1w==;20:3ihmxT1MN+pFZcjcIjI4heTGtWGC8sgvfEcc7AEZrOMqsGVev99ApQD6K+9TpgMVDAfRbDybGsMRTVCLB0jI2Q==;23:0kI0/U5z6tyEd+5hIzy23cwqX7EXyaTcKAk1XpwZH3HUu7IdJ9Jq0SSwKw9O3G0yY20wjVHftik9q3g8M+/vO2QDAZLQsOw8TrRnB+ZTJ/wBaao9Q37UTc6UzTui1OHgnZ6GFJcDEV9+5qGtQixD08uqtL1XYQqo3zS4BvO50GVlhN9A/rWLOyyZl6KosRzPdRj3nlI9pKQQV5foSfCKMQxBjbHae5oomfu1dBd9qvfPknqUOG6Qz2uulMC9eDOu
SN1PR02MB1695: X-MS-Exchange-Organization-RulesExecuted
X-OriginatorOrg: sandisk.com
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 2950
Lines: 54

On 07/15/2015 05:12 AM, Thomas Gleixner wrote:
> On Wed, 15 Jul 2015, Christoph Hellwig wrote:
>> Many years ago we decided to move setting of IRQ to core affnities to
>> userspace with the irqbalance daemon.
>>
>> These days we have systems with lots of MSI-X vector, and we have
>> hardware and subsystem support for per-CPU I/O queues in the block
>> layer, the RDMA subsystem and probably the network stack (I'm not too
>> familar with the recent developments there).  It would really help the
>> out of the box performance and experience if we could allow such
>> subsystems to bind interrupt vectors to the node that the queue is
>> configured on.
>>
>> I'd like to discuss if the rationale for moving the IRQ affinity setting
>> fully to userspace are still correct in todays world any any pitfalls
>> we'll have to learn from in irqbalanced and the old in-kernel affinity
>> code.
>
> I think setting an initial affinity is not going to create the horror
> of the old in-kernel irq balancer again. It still could be changed
> from user space and does not try to be smart by moving interrupts
> around in circles all the time.

Thanks Thomas for your feedback. But no matter whether IRQ balancing 
happens in user space or in the kernel, the following issues need to be 
addressed and have not yet been addressed today:
* irqbalanced is not aware of the relationship between MSI-X vectors.
   If e.g. two kernel drivers each allocate 24 MSI-X vectors for the
   PCIe interfaces they control irqbalanced could e.g. decide to
   associate all MSI-X vectors for the first PCIe interface with a first
   set of CPUs and the MSI-X vectors of the second PCIe interface with a
   second set of CPUs. This will result in suboptimal performance if
   these two PCIe interfaces are used alternatingly instead of
   simultaneously.
* With blk-mq and scsi-mq optimal performance can only be achieved if
   the relationship between MSI-X vector and NUMA node does not change
   over time. This is necessary to allow a blk-mq/scsi-mq driver to
   ensure that interrupts are processed on the same NUMA node as the
   node on which the data structures for a communication channel have
   been allocated. However, today there is no API that allows
   blk-mq/scsi-mq drivers and irqbalanced to exchange information
   about the relationship between MSI-X vector ranges and NUMA nodes.
   The only approach I know of that works today to define IRQ affinity
   for blk-mq/scsi-mq drivers is to disable irqbalanced and to run a
   custom script that defines IRQ affinity (see e.g. the
   spread-mlx4-ib-interrupts attachment of 
http://thread.gmane.org/gmane.linux.kernel.device-mapper.devel/21312/focus=98409).

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/