Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753214AbdGJI4i (ORCPT ); Mon, 10 Jul 2017 04:56:38 -0400 Received: from mail-oi0-f50.google.com ([209.85.218.50]:36384 "EHLO mail-oi0-f50.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750812AbdGJI4h (ORCPT ); Mon, 10 Jul 2017 04:56:37 -0400 MIME-Version: 1.0 In-Reply-To: <27b46938-ae23-9750-e0c7-09fa472d3297@arm.com> References: <1498405569-463-1-git-send-email-shankerd@codeaurora.org> <27b46938-ae23-9750-e0c7-09fa472d3297@arm.com> From: Ganapatrao Kulkarni Date: Mon, 10 Jul 2017 14:18:29 +0530 Message-ID: Subject: Re: [PATCH] irqchip: gicv3-its: Use NUMA aware memory allocation for ITS tables To: Marc Zyngier Cc: Shanker Donthineni , linux-kernel , linux-arm-kernel , Thomas Gleixner , Jason Cooper , Vikram Sethi , Jayachandran C , "ganapatrao.kulkarni@cavium.com" Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2613 Lines: 70 Hi Marc, On Mon, Jul 3, 2017 at 8:23 PM, Marc Zyngier wrote: > Hi Shanker, > > On 03/07/17 15:24, Shanker Donthineni wrote: >> Hi Marc, >> >> On 06/30/2017 03:51 AM, Marc Zyngier wrote: >>> On 30/06/17 04:01, Ganapatrao Kulkarni wrote: >>>> On Fri, Jun 30, 2017 at 8:04 AM, Ganapatrao Kulkarni >>>> wrote: >>>>> Hi Shanker, >>>>> >>>>> On Sun, Jun 25, 2017 at 9:16 PM, Shanker Donthineni >>>>> wrote: >>>>>> The NUMA node information is visible to ITS driver but not being used >>>>>> other than handling errata. This patch allocates the memory for ITS >>>>>> tables from the corresponding NUMA node using the appropriate NUMA >>>>>> aware functions. >>>> >>>> IMHO, the description would have been more constructive? >>>> >>>> "All ITS tables are mapped by default to NODE 0 memory. >>>> Adding changes to allocate memory from respective NUMA NODES of ITS devices. >>>> This will optimize tables access and avoids unnecessary inter-node traffic." >>> >>> But more importantly, I'd like to see figures showing the actual benefit >>> of this per-node allocation. Given that both of you guys have access to >>> such platforms, please show me the numbers! >>> >> >> I'll share the actual results which shows the improvement whenever >> available on our next chips. Current version of Qualcomm qdf2400 doesn't >> support multi socket configuration to capture results and share with you. >> >> Do you see any other issues with this patch apart from the performance >> improvements. I strongly believe this brings the noticeable improvement >> in numbers on systems where it has multi node memory/CPU configuration. > > I agree that it *could* show an improvement, but it very much depends on > how often the ITS misses in its caches. For this kind of patches, I want > to see two things: > > 1) It brings a measurable benefit on NUMA platforms Did some measurement of interrupt response time for LPIs and we don't see any major improvement due to caching of Tables. However, we have seen improvements of around 5%. IMO, we should merge this patch to have NUMA aware allocations and to avoid unwanted inter-node transactions. Tested-by: Ganapatrao Kulkarni > 2) it doesn't adversely impact non-NUMA systems AFAIK, no impact on non-NUMA and on single node NUMA systems. > > I can deal with (2), but I have no way of evaluating (1), mostly for the > lack of an infrastructure exercising multiple ITSs at the same time. > > Thanks, > > M. > -- > Jazz is not dead. It just smells funny... thanks Ganapat