Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755141AbdGCPPZ (ORCPT ); Mon, 3 Jul 2017 11:15:25 -0400 Received: from smtp.codeaurora.org ([198.145.29.96]:49008 "EHLO smtp.codeaurora.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755106AbdGCPPU (ORCPT ); Mon, 3 Jul 2017 11:15:20 -0400 DMARC-Filter: OpenDMARC Filter v1.3.2 smtp.codeaurora.org 300B9607A7 Authentication-Results: pdx-caf-mail.web.codeaurora.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: pdx-caf-mail.web.codeaurora.org; spf=none smtp.mailfrom=shankerd@codeaurora.org Reply-To: shankerd@codeaurora.org Subject: Re: [PATCH] irqchip: gicv3-its: Use NUMA aware memory allocation for ITS tables To: Marc Zyngier , Ganapatrao Kulkarni Cc: Jason Cooper , Vikram Sethi , linux-kernel , Jayachandran C , "ganapatrao.kulkarni@cavium.com" , Thomas Gleixner , linux-arm-kernel References: <1498405569-463-1-git-send-email-shankerd@codeaurora.org> <27b46938-ae23-9750-e0c7-09fa472d3297@arm.com> From: Shanker Donthineni Message-ID: <2a715559-3759-5bd4-1346-bc95f023093b@codeaurora.org> Date: Mon, 3 Jul 2017 10:15:15 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: <27b46938-ae23-9750-e0c7-09fa472d3297@arm.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2933 Lines: 72 Hi Marc, On 07/03/2017 09:53 AM, Marc Zyngier wrote: > Hi Shanker, > > On 03/07/17 15:24, Shanker Donthineni wrote: >> Hi Marc, >> >> On 06/30/2017 03:51 AM, Marc Zyngier wrote: >>> On 30/06/17 04:01, Ganapatrao Kulkarni wrote: >>>> On Fri, Jun 30, 2017 at 8:04 AM, Ganapatrao Kulkarni >>>> wrote: >>>>> Hi Shanker, >>>>> >>>>> On Sun, Jun 25, 2017 at 9:16 PM, Shanker Donthineni >>>>> wrote: >>>>>> The NUMA node information is visible to ITS driver but not being used >>>>>> other than handling errata. This patch allocates the memory for ITS >>>>>> tables from the corresponding NUMA node using the appropriate NUMA >>>>>> aware functions. >>>> >>>> IMHO, the description would have been more constructive? >>>> >>>> "All ITS tables are mapped by default to NODE 0 memory. >>>> Adding changes to allocate memory from respective NUMA NODES of ITS devices. >>>> This will optimize tables access and avoids unnecessary inter-node traffic." >>> >>> But more importantly, I'd like to see figures showing the actual benefit >>> of this per-node allocation. Given that both of you guys have access to >>> such platforms, please show me the numbers! >>> >> >> I'll share the actual results which shows the improvement whenever >> available on our next chips. Current version of Qualcomm qdf2400 doesn't >> support multi socket configuration to capture results and share with you. >> >> Do you see any other issues with this patch apart from the performance >> improvements. I strongly believe this brings the noticeable improvement >> in numbers on systems where it has multi node memory/CPU configuration. > > I agree that it *could* show an improvement, but it very much depends on > how often the ITS misses in its caches. For this kind of patches, I want > to see two things: > Just imagine systems with hundreds of PCI-SRIOV virtual functions and assigning some of them to virtual machines, and systems with GICv4 feature. There should be a lot of cache misses on ITS VCPU, DEVICE and COLLECTION lookups. And also VLPI patches that you've posted for comments are forcing to use VLPI feature for each VM irrespective of pass-through device assignment. > 1) It brings a measurable benefit on NUMA platforms > 2) it doesn't adversely impact non-NUMA systems > It should not affect the ITS hardware behavior non-NUMA based system since software always allocate memory from a single (default) NUMA node. > I can deal with (2), but I have no way of evaluating (1), mostly for the > lack of an infrastructure exercising multiple ITSs at the same time. > Agree with you, but it takes some time for me to provide the test results. > Thanks, > > M. > -- Shanker Donthineni Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.