Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757133Ab0GRSwu (ORCPT ); Sun, 18 Jul 2010 14:52:50 -0400 Received: from mail-pz0-f46.google.com ([209.85.210.46]:55403 "EHLO mail-pz0-f46.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757097Ab0GRSwt (ORCPT ); Sun, 18 Jul 2010 14:52:49 -0400 DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=dzy1xkewTQznbGdnc2g0luJc6pz0m96i5uxM6/gWZ92fgH1fNMq9z63EyFhdl0/tvj UdKHHZrawGnxeRDz3As0npO1/IFGazMSzZzXM6ihHhpE3+oho13wk2Nr5yt1kXZXqK7x b2m/+FCVIsjwpPCR1RDw/msI+POb8iIfB1RAA= MIME-Version: 1.0 In-Reply-To: <4C434A19.9050207@linux.vnet.ibm.com> References: <4C37C365.6090508@gmail.com> <4C434A19.9050207@linux.vnet.ibm.com> Date: Sun, 18 Jul 2010 11:52:48 -0700 Message-ID: Subject: Re: Interrupt Affinity in SMP From: Bryan Hundven To: Ciju Rajan K Cc: linux-kernel@vger.kernel.org, Robert Hancock , mchehab@redhat.com Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3062 Lines: 105 On Sun, Jul 18, 2010 at 11:38 AM, Ciju Rajan K wrote: > Bryan Hundven wrote: >> >> On Sat, Jul 10, 2010 at 6:20 PM, Robert Hancock >> wrote: >> >>> >>> On Sat, Jul 10, 2010 at 1:46 PM, Bryan Hundven >>> wrote: >>> >>>> >>>> I was able to set eth0 and it's TxRx queues to cpu1, but it is my >>>> understanding that 0xFFFFFFFF should distribute the interrupts across >>>> all >>>> cpus, much like LOC in my output of /proc/interrupts. >>>> >>>> I don't have access to the computer this weekend, but I will provide >>>> more >>>> info on Monday. >>>> >>> >>> That may be chipset dependent, I don't think all chipsets have the >>> ability to distribute the interrupts like that. Round-robin interrupt >>> distribution for a given handler isn't optimal for performance anyway >>> since it causes the relevant cache lines for the interrupt handler to >>> be ping-ponged between the different CPUs. >>> >>> >>>> >>>> -bryan >>>> >>>> On Jul 9, 2010 5:48 PM, "Robert Hancock" wrote: >>>> >>>> On 07/09/2010 04:59 PM, Bryan Hundven wrote: >>>> >>>>> >>>>> Mauro, list, >>>>> >>>>> (please CC me in replies, I am not... >>>>> >>>> >>>> Tried changing these files to exclude CPU0? >>>> >>>> Have you tried running the irqbalance daemon? That's what you likely >>>> want to >>>> be doing anyway.. >>>> >>>> >>>>> >>>>> =====8<=====8<=====8<=====8<=====8<=====8<=====8<=====8<=====8<===== >>>>> >>>>> =====8<=====8<=====8<==... >>>>> >> >> Please see the two attached examples. >> >> Notice on the 5410 example how we start with the affinity set to 0xff, >> and change it to 0xf0. >> This should spread the interrupts over the last 4 cores of this quad >> core - dual processor system. >> >> Also notice on the 5645 example, with the same commands we start with >> 0xffffff and change to 0xfff000 to spread the interrupts over the last >> 12 cores, but only the first of the last twelve cores receive >> interrupts. >> >> This is the inconsistency I was trying to explain before. >> > > What was the status of irqbalance daemon? Was it turned on? If it is > running, there is a chance that the interrupt count is within the threshold > limit and interrupts are not being routed to the other core. irqbalance daemon was not running on either setup. > Could you also try with increasing the interrupt load and see if the > distribution is happening among the cores? We use spirent testcenter l2/l3 test equipment and pushed 100% throughput with the same distribution. Nothing changed. This isn't affecting just ethernet drivers. I have also seen the same issues with hardware encryption devices and other hardware that gets a software interrupt. --Bryan > > -Ciju >> >> --Bryan >> > > -- Bryan Hundven bryanhundven@gmail.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/