Received: by 2002:ac2:464d:0:0:0:0:0 with SMTP id s13csp3290540lfo; Mon, 23 May 2022 00:50:54 -0700 (PDT) X-Google-Smtp-Source: ABdhPJx9pxU/pYLqttiW/YdwwxQPq8mjzTqm7DFYaWiahrocE63yxniDU8Ku6X4Lyj2fcJ/vcTWJ X-Received: by 2002:a17:902:a9cc:b0:161:3bf4:9047 with SMTP id b12-20020a170902a9cc00b001613bf49047mr21393836plr.137.1653292253868; Mon, 23 May 2022 00:50:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1653292253; cv=none; d=google.com; s=arc-20160816; b=BT5f1384JDVMjrLvShquwrQgCH2XLAmyCLkyMUCTMW4g0OiNpHK/7g0pAM51ifMISH nE3dXZ0U1DwJKrj4IljqDbas2xDjDN2O7fd2jmtSv9BCPHUXs6NL1fqFmygfGkcDYD9g XvfYZX3zzPDNcH7mRbkjLuggxbJALzUNaMzszSw9beWNPBANL9qqp3vHaq1faoRUJRWF zFF9fi3EFJjy+3xynDQDNEfkehkUqLWUs8jid3vkQsml82hRBt37swkoUEWl5ihQ2KJm HctlCI6MmtCTMdRDkbKjPkAzQw4ba2O8RUSdDE2CccdEhr5X0ASwc2RuEX0KlAaIheVg hoew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:in-reply-to:from :references:cc:to:content-language:subject:user-agent:mime-version :date:message-id:dkim-signature; bh=eFxAkTiFYx/27rNrxiNuMe1vTCHfN4tVz87m6TJA89U=; b=zS/I4FEbBgPF2fMAtZgGbBZ0VJ3+YFYbiQM+bt6iZPIhmwK78VYGI0jMR6hD2mNKD/ MMs8TQFrC1D7fhqXtUdGoxLmpaXAADFW1qGIfNxVdxw8UCD01OPeoYUfHK47bq3bSZup hxhoM6nKYo2sEoueDTQQsf5nfEbUZTetrzIbqWznY90fdVeigg85H+7LRwxLUgJ/2Flk begksEsFYR5MefAerxL6SLkrfaZb2gjf1fITnrF37DFbwLpevqFtaXVZYOvXEVtP4hOJ JM0vfM7v4nR9jf1/Z+RfDw/cy10N6U2ebOjgnponm09k5+1JMm38UpVpQZyTxm5icka6 rYZA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ECZCR2Wz; spf=softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net. [23.128.96.19]) by mx.google.com with ESMTPS id lr1-20020a17090b4b8100b001d9654a4447si14788400pjb.132.2022.05.23.00.50.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 May 2022 00:50:53 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ECZCR2Wz; spf=softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 41605222431; Sun, 22 May 2022 23:54:41 -0700 (PDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240674AbiESPRH (ORCPT + 99 others); Thu, 19 May 2022 11:17:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53094 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240661AbiESPRB (ORCPT ); Thu, 19 May 2022 11:17:01 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 57DA65FF09 for ; Thu, 19 May 2022 08:16:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1652973418; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eFxAkTiFYx/27rNrxiNuMe1vTCHfN4tVz87m6TJA89U=; b=ECZCR2WzbqgSWgTXmvLZNNqXQsvlIxYsSZd9y54f5Je7rPrStf5I9irxtT262XG5sYutWy sTT9VOzqo1dHM7r6BI5fA3h5Ds0uCM2YPbPM3csk2G+dysvHmW8LJ1V8p7mX8jdruhDroS uYJD1d4JQnPAW6iy6Nbn2/ON6SDqlFQ= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-228-Ja2n8FvbONii9obOO1aGBg-1; Thu, 19 May 2022 11:16:57 -0400 X-MC-Unique: Ja2n8FvbONii9obOO1aGBg-1 Received: by mail-qt1-f200.google.com with SMTP id l7-20020a05622a174700b002f3c49f49ffso4489987qtk.15 for ; Thu, 19 May 2022 08:16:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :content-language:to:cc:references:from:in-reply-to :content-transfer-encoding; bh=eFxAkTiFYx/27rNrxiNuMe1vTCHfN4tVz87m6TJA89U=; b=WntLrVIqI9kZVw4JHpUoCIAV+ebpv0tRy5FTaysCKS4xTOqeym6CXoNxw4AjNcR12e IQGBQuV2rwDi1dULmZ45KeYu9KzFmJD9kshTCtSNYg/BDaTVpT8RGrvIux36QuC2uK/w WPgPQ/Skid0Cjeads+aDyf03KaXZRya+tv9hZMAgYNq2ZZEQCMTEb1Xv7FkNUy259E9l oDTp/JfelV8JYbGpYP9V7Wm2NB4aLqGR47QO2XjkD9WmAxOTYzoOaYWmmyTDng0AYsaV fcKHhQKu1ZkoLTaDaA8rlr+YRYm1aXy7poUTFaB7p4Efwj4p/KvkFekrlzFtlW7R8L11 HHIw== X-Gm-Message-State: AOAM533X2vgyAgOdQfTiBzw7dwLMf+IcANE89Wdq40ba+82NFPe0eZgH ayXYnhdK8GJuM+xQAl+92UjkZBo5pt3t7YXIGnkuQMk5D8mLIpvUX27uZ1D530ERfzrwpFXhHlJ ypDQmYX0PgEWg/tjj0VBoUNKM X-Received: by 2002:a05:622a:15cb:b0:2f3:f90f:1939 with SMTP id d11-20020a05622a15cb00b002f3f90f1939mr4332869qty.374.1652973416660; Thu, 19 May 2022 08:16:56 -0700 (PDT) X-Received: by 2002:a05:622a:15cb:b0:2f3:f90f:1939 with SMTP id d11-20020a05622a15cb00b002f3f90f1939mr4332845qty.374.1652973416375; Thu, 19 May 2022 08:16:56 -0700 (PDT) Received: from [192.168.1.86] (pool-173-48-131-43.bstnma.fios.verizon.net. [173.48.131.43]) by smtp.gmail.com with ESMTPSA id c190-20020ae9edc7000000b0069fc13ce1e3sm1347636qkg.20.2022.05.19.08.16.54 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 19 May 2022 08:16:55 -0700 (PDT) Message-ID: <32e5a3b7-9294-bbd5-0ae4-b5c04eb4e0e6@redhat.com> Date: Thu, 19 May 2022 11:16:53 -0400 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.9.0 Subject: Re: [PATCH v8 0/4] perf: arm-spe: Decode SPE source and use for perf c2c Content-Language: en-US To: Leo Yan , Arnaldo Carvalho de Melo Cc: Ali Saidi , linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, linux-arm-kernel@lists.infradead.org, german.gomez@arm.com, benh@kernel.crashing.org, Nick.Forrington@arm.com, alexander.shishkin@linux.intel.com, andrew.kilroy@arm.com, james.clark@arm.com, john.garry@huawei.com, Jiri Olsa , kjain@linux.ibm.com, lihuafei1@huawei.com, mark.rutland@arm.com, mathieu.poirier@linaro.org, mingo@redhat.com, namhyung@kernel.org, peterz@infradead.org, will@kernel.org References: <20220517020326.18580-1-alisaidi@amazon.com> <20220518041630.GD402837@leoy-ThinkPad-X240s> From: Joe Mario In-Reply-To: <20220518041630.GD402837@leoy-ThinkPad-X240s> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-3.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,NICE_REPLY_A,RDNS_NONE,SPF_HELO_NONE, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 5/18/22 12:16 AM, Leo Yan wrote: > Hi Joe, > > On Tue, May 17, 2022 at 06:20:03PM -0300, Arnaldo Carvalho de Melo wrote: >> Em Tue, May 17, 2022 at 02:03:21AM +0000, Ali Saidi escreveu: >>> When synthesizing data from SPE, augment the type with source information >>> for Arm Neoverse cores so we can detect situtions like cache line >>> contention and transfers on Arm platforms. >>> >>> This changes enables future changes to c2c on a system with SPE where lines that >>> are shared among multiple cores show up in perf c2c output. >>> >>> Changes is v9: >>> * Change reporting of remote socket data which should make Leo's upcomping >>> patch set for c2c make sense on multi-socket platforms >> >> Hey, >> >> Joe Mario, who is one of 'perf c2c' authors asked me about some >> git tree he could clone from for both building the kernel and >> tools/perf/ so that he could do tests, can you please provide that? > > I have uploaded the latest patches for enabling 'perf c2c' on Arm SPE > on the repo: > > https://git.linaro.org/people/leo.yan/linux-spe.git branch: perf_c2c_arm_spe_peer_v3 > > Below are the quick notes for build the kernel with enabling Arm SPE: > > $ git clone -b perf_c2c_arm_spe_peer_v3 https://git.linaro.org/people/leo.yan/linux-spe.git > > Or > > $ git clone -b perf_c2c_arm_spe_peer_v3 ssh://git@git.linaro.org/people/leo.yan/linux-spe.git > > $ cd linux-spe > > # Build kernel > $ make defconfig > $ ./scripts/config -e CONFIG_PID_IN_CONTEXTIDR > $ ./scripts/config -e CONFIG_ARM_SPE_PMU > $ make Image > > # Build perf > $ cd tools/perf > $ make VF=1 DEBUG=1 > > When boot the kernel, please add option "kpti=off" in kernel command > line, you might need to update grub menu for this. > > Please feel free let us know if anything is not clear for you. > > Thank you, > Leo > Hi Leo: Thanks for getting this working on ARM. I do have a few comments. I built and ran this on a ARM Neoverse-N1 system with 2 numa nodes. Comment 1: When I run "perf c2c report", the "Node" field is marked "N/A". It's supposed to show the numa node where the data address for the cacheline resides. That's important both to see what node hot data resides on and if that data is getting lots of cross-numa accesses. Comment 2: I'm assuming you're identifying the contended cachelines using the "peer" load response, which indicates the load was resolved from a "peer" cpu's cacheline. Please confirm. If that's true, is it possible to identify if that "peer" response was on the local or remote numa node? I ask because being able to identify both local and remote HitM's on Intel X86_64 has been quite valuable. That's because remote HitM's are costly and because it helps the viewer see if they need to optimize their cpu affinity or what node their hot data resides on. Last Comment: There's a row in the Pareto table that has incorrect column alignment. Look at row 80 below in the truncated snipit of output. It has an extra field inserted in it at the beginning. I also show what the corrected output should look like. Incorrect row 80: 71 ================================================= 72 Shared Cache Line Distribution Pareto 73 ================================================= 74 # 75 # ----- HITM ----- Snoop ------- Store Refs ------ ------- CL -------- 76 # RmtHitm LclHitm Peer L1 Hit L1 Miss N/A Off Node PA cnt Code address 77 # ....... ....... ....... ....... ....... ....... ..... .... ...... .................. 78 # 79 ------------------------------------------------------------------------------- 80 0 0 0 4648 0 0 11572 0x422140 81 ------------------------------------------------------------------------------- 82 0.00% 0.00% 0.00% 0.00% 0.00% 44.47% 0x0 N/A 0 0x400ce8 83 0.00% 0.00% 10.26% 0.00% 0.00% 0.00% 0x0 N/A 0 0x400e48 84 0.00% 0.00% 0.00% 0.00% 0.00% 55.53% 0x0 N/A 0 0x400e54 85 0.00% 0.00% 89.74% 0.00% 0.00% 0.00% 0x8 N/A 0 0x401038 Corrected row 80: 71 ================================================= 72 Shared Cache Line Distribution Pareto 73 ================================================= 74 # 75 # ----- HITM ----- Snoop ------- Store Refs ----- ------- CL -------- 76 # RmtHitm LclHitm Peer L1 Hit L1 Miss N/A Off Node PA cnt Code address 77 # ....... ....... ....... ....... ....... ...... ..... .... ...... .................. 78 # 79 ------------------------------------------------------------------------------- 80 0 0 4648 0 0 11572 0x422140 81 ------------------------------------------------------------------------------- 82 0.00% 0.00% 0.00% 0.00% 0.00% 44.47% 0x0 N/A 0 0x400ce8 83 0.00% 0.00% 10.26% 0.00% 0.00% 0.00% 0x0 N/A 0 0x400e48 84 0.00% 0.00% 0.00% 0.00% 0.00% 55.53% 0x0 N/A 0 0x400e54 85 0.00% 0.00% 89.74% 0.00% 0.00% 0.00% 0x8 N/A 0 0x401038 Thanks again for doing this. Joe