Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753102AbdCWLwy (ORCPT ); Thu, 23 Mar 2017 07:52:54 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:37146 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751228AbdCWLww (ORCPT ); Thu, 23 Mar 2017 07:52:52 -0400 Date: Thu, 23 Mar 2017 17:22:37 +0530 From: Gautham R Shenoy To: Madhavan Srinivasan Cc: mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Hemant Kumar , "Gautham R . Shenoy" , Balbir Singh , Benjamin Herrenschmidt , Paul Mackerras , Anton Blanchard , Sukadev Bhattiprolu , Michael Neuling , Stewart Smith , Daniel Axtens , Stephane Eranian , Anju T Sudhakar Subject: Re: [PATCH v5 06/13] powerpc/perf: IMC pmu cpumask and cpu hotplug support Reply-To: ego@linux.vnet.ibm.com References: <1489649707-8021-1-git-send-email-maddy@linux.vnet.ibm.com> <1489649707-8021-7-git-send-email-maddy@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1489649707-8021-7-git-send-email-maddy@linux.vnet.ibm.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-TM-AS-GCONF: 00 x-cbid: 17032311-0020-0000-0000-00000BA0415F X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00006834; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000206; SDB=6.00837765; UDB=6.00411890; IPR=6.00615528; BA=6.00005233; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00014773; XFM=3.00000013; UTC=2017-03-23 11:52:49 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17032311-0021-0000-0000-00005B1694DD Message-Id: <20170323115237.GA4897@in.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-03-23_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=2 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1702020001 definitions=main-1703230106 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 8637 Lines: 262 Hi Hemant, Maddy, On Thu, Mar 16, 2017 at 01:05:00PM +0530, Madhavan Srinivasan wrote: > From: Hemant Kumar > > Adds cpumask attribute to be used by each IMC pmu. Only one cpu (any > online CPU) from each chip for nest PMUs is designated to read counters. > > On CPU hotplug, dying CPU is checked to see whether it is one of the > designated cpus, if yes, next online cpu from the same chip (for nest > units) is designated as new cpu to read counters. For this purpose, we > introduce a new state : CPUHP_AP_PERF_POWERPC_NEST_ONLINE. > > Cc: Gautham R. Shenoy > Cc: Balbir Singh > Cc: Benjamin Herrenschmidt > Cc: Paul Mackerras > Cc: Anton Blanchard > Cc: Sukadev Bhattiprolu > Cc: Michael Neuling > Cc: Stewart Smith > Cc: Daniel Axtens > Cc: Stephane Eranian > Cc: Anju T Sudhakar > Signed-off-by: Hemant Kumar > Signed-off-by: Madhavan Srinivasan > --- > arch/powerpc/include/asm/opal-api.h | 3 +- > arch/powerpc/include/asm/opal.h | 3 + > arch/powerpc/perf/imc-pmu.c | 163 ++++++++++++++++++++++++- > arch/powerpc/platforms/powernv/opal-wrappers.S | 1 + > include/linux/cpuhotplug.h | 1 + > 5 files changed, 169 insertions(+), 2 deletions(-) > > diff --git a/arch/powerpc/include/asm/opal-api.h b/arch/powerpc/include/asm/opal-api.h > index a0aa285869b5..e1c3d4837857 100644 > --- a/arch/powerpc/include/asm/opal-api.h > +++ b/arch/powerpc/include/asm/opal-api.h > @@ -168,7 +168,8 @@ > #define OPAL_INT_SET_MFRR 125 > #define OPAL_PCI_TCE_KILL 126 > #define OPAL_NMMU_SET_PTCR 127 > -#define OPAL_LAST 127 > +#define OPAL_NEST_IMC_COUNTERS_CONTROL 145 > +#define OPAL_LAST 145 > > /* Device tree flags */ > > diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h > index 1ff03a6da76e..d93d08204243 100644 > --- a/arch/powerpc/include/asm/opal.h > +++ b/arch/powerpc/include/asm/opal.h > @@ -227,6 +227,9 @@ int64_t opal_pci_tce_kill(uint64_t phb_id, uint32_t kill_type, > uint64_t dma_addr, uint32_t npages); > int64_t opal_nmmu_set_ptcr(uint64_t chip_id, uint64_t ptcr); > > +int64_t opal_nest_imc_counters_control(uint64_t mode, uint64_t value1, > + uint64_t value2, uint64_t value3); > + > /* Internal functions */ > extern int early_init_dt_scan_opal(unsigned long node, const char *uname, > int depth, void *data); > diff --git a/arch/powerpc/perf/imc-pmu.c b/arch/powerpc/perf/imc-pmu.c > index f6f1ef9f56af..e46ff6d2a584 100644 > --- a/arch/powerpc/perf/imc-pmu.c > +++ b/arch/powerpc/perf/imc-pmu.c > @@ -16,6 +16,7 @@ > +static int ppc_nest_imc_cpu_online(unsigned int cpu) > +{ I take it that 'cpu' is coming online. > + int nid, fcpu, ncpu; > + struct cpumask *l_cpumask, tmp_mask; > + > + /* Fint the cpumask of this node */ > + nid = cpu_to_node(cpu); > + l_cpumask = cpumask_of_node(nid); > + > + /* > + * If any of the cpu from this node is already present in the mask, > + * just return, if not, then set this cpu in the mask. > + */ > + if (!cpumask_and(&tmp_mask, l_cpumask, &nest_imc_cpumask)) { In this case, none of the cpus in the node are in the mask. So we set and this cpu in the imc cpumask and return. > + cpumask_set_cpu(cpu, &nest_imc_cpumask); > + return 0; > + } But this case implies that there is already a CPU from the node which is in the imc_cpumask. As per the comment above, we are supposed to just return. So why are we doing the following ? Either the comment above is incorrect or I am missing something here. > + > + fcpu = cpumask_first(l_cpumask); > + ncpu = cpumask_next(cpu, l_cpumask); > + if (cpu == fcpu) { > + if (cpumask_test_and_clear_cpu(ncpu, &nest_imc_cpumask)) { > + cpumask_set_cpu(cpu, &nest_imc_cpumask); > + nest_change_cpu_context(ncpu, cpu); > + } > + } It seems that we want to set only the smallest online cpu in the node in the nest_imc_cpumask. So, if the newly onlined cpu is the smallest, we replace the previous representative with cpu. So, the comment above needs to be fixed. > + > + return 0; > +} > + > +static int ppc_nest_imc_cpu_offline(unsigned int cpu) > +{ > + int nid, target = -1; > + struct cpumask *l_cpumask; > + > + /* > + * Check in the designated list for this cpu. Dont bother > + * if not one of them. > + */ > + if (!cpumask_test_and_clear_cpu(cpu, &nest_imc_cpumask)) > + return 0; > + > + /* > + * Now that this cpu is one of the designated, > + * find a next cpu a) which is online and b) in same chip. > + */ > + nid = cpu_to_node(cpu); > + l_cpumask = cpumask_of_node(nid); > + target = cpumask_next(cpu, l_cpumask); > + > + /* > + * Update the cpumask with the target cpu and > + * migrate the context if needed > + */ > + if (target >= 0 && target <= nr_cpu_ids) { > + cpumask_set_cpu(target, &nest_imc_cpumask); > + nest_change_cpu_context(cpu, target); > + } > + return 0; > +} > + > +static int nest_pmu_cpumask_init(void) > +{ > + const struct cpumask *l_cpumask; > + int cpu, nid; > + int *cpus_opal_rc; > + > + if (!cpumask_empty(&nest_imc_cpumask)) > + return 0; > + > + /* > + * Nest PMUs are per-chip counters. So designate a cpu > + * from each chip for counter collection. > + */ > + for_each_online_node(nid) { > + l_cpumask = cpumask_of_node(nid); > + > + /* designate first online cpu in this node */ > + cpu = cpumask_first(l_cpumask); > + cpumask_set_cpu(cpu, &nest_imc_cpumask); > + } > + What happens if a cpu in nest_imc_cpumask goes offline at this point ? Is that possible ? > + /* > + * Memory for OPAL call return value. > + */ > + cpus_opal_rc = kzalloc((sizeof(int) * nr_cpu_ids), GFP_KERNEL); > + if (!cpus_opal_rc) > + goto fail; > + > + /* Initialize Nest PMUs in each node using designated cpus */ > + on_each_cpu_mask(&nest_imc_cpumask, (smp_call_func_t)nest_init, > + (void *)cpus_opal_rc, 1); > + > + /* Check return value array for any OPAL call failure */ > + for_each_cpu(cpu, &nest_imc_cpumask) { > + if (cpus_opal_rc[cpu]) > + goto fail; > + } > + > + cpuhp_setup_state(CPUHP_AP_PERF_POWERPC_NEST_ONLINE, > + "POWER_NEST_IMC_ONLINE", > + ppc_nest_imc_cpu_online, > + ppc_nest_imc_cpu_offline); > + > + return 0; > + > +fail: > + return -ENODEV; > +} > + > static int nest_imc_event_init(struct perf_event *event) > { > int chip_id; > @@ -63,7 +218,7 @@ static int nest_imc_event_init(struct perf_event *event) > chip_id = topology_physical_package_id(event->cpu); > pcni = &nest_perchip_info[chip_id]; > event->hw.event_base = pcni->vbase[config/PAGE_SIZE] + > - (config & ~PAGE_MASK); > + (config & ~PAGE_MASK); > > return 0; > } > @@ -122,6 +277,7 @@ static int update_pmu_ops(struct imc_pmu *pmu) > pmu->pmu.stop = imc_event_stop; > pmu->pmu.read = imc_perf_event_update; > pmu->attr_groups[1] = &imc_format_group; > + pmu->attr_groups[2] = &imc_pmu_cpumask_attr_group; > pmu->pmu.attr_groups = pmu->attr_groups; > > return 0; > @@ -189,6 +345,11 @@ int init_imc_pmu(struct imc_events *events, int idx, > { > int ret = -ENODEV; > > + /* Add cpumask and register for hotplug notification */ > + ret = nest_pmu_cpumask_init(); > + if (ret) > + return ret; > + > ret = update_events_in_group(events, idx, pmu_ptr); > if (ret) > goto err_free; > diff --git a/arch/powerpc/platforms/powernv/opal-wrappers.S b/arch/powerpc/platforms/powernv/opal-wrappers.S > index da8a0f7a035c..b7208d8e6cc0 100644 > --- a/arch/powerpc/platforms/powernv/opal-wrappers.S > +++ b/arch/powerpc/platforms/powernv/opal-wrappers.S > @@ -301,3 +301,4 @@ OPAL_CALL(opal_int_eoi, OPAL_INT_EOI); > OPAL_CALL(opal_int_set_mfrr, OPAL_INT_SET_MFRR); > OPAL_CALL(opal_pci_tce_kill, OPAL_PCI_TCE_KILL); > OPAL_CALL(opal_nmmu_set_ptcr, OPAL_NMMU_SET_PTCR); > +OPAL_CALL(opal_nest_imc_counters_control, OPAL_NEST_IMC_COUNTERS_CONTROL); > diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h > index 62d240e962f0..cfb0cedc72af 100644 > --- a/include/linux/cpuhotplug.h > +++ b/include/linux/cpuhotplug.h > @@ -136,6 +136,7 @@ enum cpuhp_state { > CPUHP_AP_PERF_ARM_CCI_ONLINE, > CPUHP_AP_PERF_ARM_CCN_ONLINE, > CPUHP_AP_PERF_ARM_L2X0_ONLINE, > + CPUHP_AP_PERF_POWERPC_NEST_ONLINE, > CPUHP_AP_PERF_ARM_QCOM_L2_ONLINE, > CPUHP_AP_WORKQUEUE_ONLINE, > CPUHP_AP_RCUTREE_ONLINE, > -- > 2.7.4 >