Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757178Ab0GRTWt (ORCPT ); Sun, 18 Jul 2010 15:22:49 -0400 Received: from e23smtp07.au.ibm.com ([202.81.31.140]:45117 "EHLO e23smtp07.au.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757153Ab0GRTWs (ORCPT ); Sun, 18 Jul 2010 15:22:48 -0400 Message-ID: <4C435481.5080506@linux.vnet.ibm.com> Date: Mon, 19 Jul 2010 00:52:41 +0530 From: Ciju Rajan K User-Agent: Thunderbird 2.0.0.21 (X11/20090320) MIME-Version: 1.0 To: Bryan Hundven CC: linux-kernel@vger.kernel.org, Robert Hancock , mchehab@redhat.com Subject: Re: Interrupt Affinity in SMP References: <4C37C365.6090508@gmail.com> <4C434A19.9050207@linux.vnet.ibm.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3456 Lines: 114 Bryan Hundven wrote: > On Sun, Jul 18, 2010 at 11:38 AM, Ciju Rajan K wrote: > >> Bryan Hundven wrote: >> >>> On Sat, Jul 10, 2010 at 6:20 PM, Robert Hancock >>> wrote: >>> >>> >>>> On Sat, Jul 10, 2010 at 1:46 PM, Bryan Hundven >>>> wrote: >>>> >>>> >>>>> I was able to set eth0 and it's TxRx queues to cpu1, but it is my >>>>> understanding that 0xFFFFFFFF should distribute the interrupts across >>>>> all >>>>> cpus, much like LOC in my output of /proc/interrupts. >>>>> >>>>> I don't have access to the computer this weekend, but I will provide >>>>> more >>>>> info on Monday. >>>>> >>>>> >>>> That may be chipset dependent, I don't think all chipsets have the >>>> ability to distribute the interrupts like that. Round-robin interrupt >>>> distribution for a given handler isn't optimal for performance anyway >>>> since it causes the relevant cache lines for the interrupt handler to >>>> be ping-ponged between the different CPUs. >>>> >>>> >>>> >>>>> -bryan >>>>> >>>>> On Jul 9, 2010 5:48 PM, "Robert Hancock" wrote: >>>>> >>>>> On 07/09/2010 04:59 PM, Bryan Hundven wrote: >>>>> >>>>> >>>>>> Mauro, list, >>>>>> >>>>>> (please CC me in replies, I am not... >>>>>> >>>>>> >>>>> Tried changing these files to exclude CPU0? >>>>> >>>>> Have you tried running the irqbalance daemon? That's what you likely >>>>> want to >>>>> be doing anyway.. >>>>> >>>>> >>>>> >>>>>> =====8<=====8<=====8<=====8<=====8<=====8<=====8<=====8<=====8<===== >>>>>> >>>>>> =====8<=====8<=====8<==... >>>>>> >>>>>> >>> Please see the two attached examples. >>> >>> Notice on the 5410 example how we start with the affinity set to 0xff, >>> and change it to 0xf0. >>> This should spread the interrupts over the last 4 cores of this quad >>> core - dual processor system. >>> >>> Also notice on the 5645 example, with the same commands we start with >>> 0xffffff and change to 0xfff000 to spread the interrupts over the last >>> 12 cores, but only the first of the last twelve cores receive >>> interrupts. >>> >>> This is the inconsistency I was trying to explain before. >>> >>> >> What was the status of irqbalance daemon? Was it turned on? If it is >> running, there is a chance that the interrupt count is within the threshold >> limit and interrupts are not being routed to the other core. >> > > irqbalance daemon was not running on either setup. > > >> Could you also try with increasing the interrupt load and see if the >> distribution is happening among the cores? >> > > We use spirent testcenter l2/l3 test equipment and pushed 100% > throughput with the same distribution. Nothing changed. > In the example that you have given, I could see just 7 interrupts after 15 seconds. So thought of checking it. Let me try to see this problem locally. -Ciju > This isn't affecting just ethernet drivers. I have also seen the same > issues with hardware encryption devices and other hardware that gets a > software interrupt. > > --Bryan > > >> -Ciju >> >>> --Bryan >>> >>> >> > > > > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/