Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Huang Shijie <shijie@os.amperecomputing.com>
To:     maz@kernel.org
Cc:     oliver.upton@linux.dev, james.morse@arm.com,
        suzuki.poulose@arm.com, yuzenghui@huawei.com,
        catalin.marinas@arm.com, will@kernel.org,
        linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev,
        linux-kernel@vger.kernel.org, patches@amperecomputing.com,
        zwang@amperecomputing.com,
        Huang Shijie <shijie@os.amperecomputing.com>
Subject: [PATCH v2] KVM/arm64: reconfigurate the event filters for guest context
Date:   Thu, 10 Aug 2023 15:29:06 +0800
Message-Id: <20230810072906.4007-1-shijie@os.amperecomputing.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?uHmxR/qS5mIovXvH5AbSbED1EmCyPMjDwaJoRHFN/O5/CJmivBEcdRJiyzxA?=
 =?us-ascii?Q?m2jaFY1+7WscpmWiYiWf+AfrxW1DMV/mTYqGUiZ+O8tVcEG7OUN9DLjgMJMX?=
 =?us-ascii?Q?dFpWFE4IbM1JYGLVq2U/xAKrHEJI3Sizcd60g1KN2NDpmxI36tJDeqK7+QF5?=
 =?us-ascii?Q?Q+HzvJGbIcy+EWVqyYuM55/QDYrivBEO8/p5OgQ3plzwG9zApZ0YKsbeu9sQ?=
 =?us-ascii?Q?EGG1eqAPSOF6D9P85VU6zG/mtr1S5sqjm84KpVMk/8BMqLLPkIO0Jg4mmKcp?=
 =?us-ascii?Q?R233hh11IEm6OaCxOozO7QbviWiim8k/8YDbzrSih/XeG/H2eTmjtH4UdONy?=
 =?us-ascii?Q?CuFhMXmE5oHYNevmKEc4uRtAL7lvaMdrlNr6TQ7PKMvtEodx8IVj/D9FLstV?=
 =?us-ascii?Q?324ZjkNHmxutRC2keBw1NF75SJ8VJ9/KY+K8ysL7jJrq6AdyMM6QVCha8AMu?=
 =?us-ascii?Q?lxGrMI0MjFtP+FbtEeijBJzdehInwpefyZJ5k7jiJAemc/YAAr3UP+TrDKWZ?=
 =?us-ascii?Q?zVn0XzelILjz4yMmaimRnpSQ4rVOgX/DbAiJi3fuZ/SOECIcBrK8/rNufp2v?=
 =?us-ascii?Q?+fE5ipJyVMDf4pY2EYp8L0QUeum2cBgsbqsHzpTvpQadWElSAP9v1BUs/7h6?=
 =?us-ascii?Q?4xnaQrrJVKRRtY/i00L2hgzRkVuUKd9+alhJ4lI+KXiAq8YtIZo+B/Nuoikj?=
 =?us-ascii?Q?iesFNhnij1UG6mppMBzFGSRaOXn4Ycu79xm4fo8laPUau6lgqKrAEfJHr5uT?=
 =?us-ascii?Q?+Me/sGtzQD8ri3iuluG/qcfvVZSWhD/YtQjI8GhUOX3LoMaY6vSO4Uayygoh?=
 =?us-ascii?Q?JuvLVQgwZX3hetTLRtrSroZeOP+WG8TIPStc472x5Po9MKgkU2uVv7dsegtx?=
 =?us-ascii?Q?riv8tzwcldkXPkS7/3yMuKN+Tfh6Vp5ejbont3s4Q56MPsMqvtBuKquseXKZ?=
 =?us-ascii?Q?ZsVuWj9UE44bI+ZT9J5TlkPE1gF1/7OF+eoehyjB6jB/MZfG2JU/phqCyPvx?=
 =?us-ascii?Q?3dfr/bGu+E+qAopToAAJiKBt+bF2Piqq1lNSdVdqO0TVg5sbSVH2W1k8/nT3?=
 =?us-ascii?Q?Atq5Wu5ijU/w7kQXWZNlDI84Xa+1DEcuN+lg4OEpIL6n60JlQiS8/0DxXDnr?=
 =?us-ascii?Q?SwzW4sqZWJgIwL8T9LKTxJLuW5npGR3c/sjJ8bsY3oySLXvsYlg7/HAiy4m6?=
 =?us-ascii?Q?aLuE5196UGZ6ucBzN8njCroChMX8d1/xOHyJo8IKMVDwXu9dLkypFP9zNc8N?=
 =?us-ascii?Q?d4fZZoqB0kxq3tPSafkJQjq1cZ7a3zLdbvTPOZsepjAffrZrElBlNLu9cG3y?=
 =?us-ascii?Q?fu3j/KteAZhaeveF7Q81uazjPTRKo0Cl7Qb9Z23khEM4bTS8mUwK9R1bZgaB?=
 =?us-ascii?Q?+JWAdZW74a+01hQQRi12QlseDq4oP4Sqai+MsETVP6iIDZ2wPS+Z7B0v8OfT?=
 =?us-ascii?Q?ldoUrh1/F49vYJ44g3mDI9GgnUqbZBLcvNwQhrJT9w/mvyq/nRyXPrO4HLPK?=
 =?us-ascii?Q?ijeKsK4BNgvZsS12HqMQavsPaYEECMvp343uJl0vmNru7ueMCJJTKObCSTSM?=
 =?us-ascii?Q?61FheghoRyVsnNG24otG0zjkPQuRj2hqzG/wmH24fDAGXbmrodAf2GERL/8S?=
 =?us-ascii?Q?NdH9ORoZOSRhkBAqYr/nGmg=3D?=
X-OriginatorOrg: os.amperecomputing.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 06d1d45d-6c8b-4b32-616e-08db99742081
X-MS-Exchange-CrossTenant-AuthSource: DM8PR01MB6824.prod.exchangelabs.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Aug 2023 07:33:45.2980
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: h0r3+COfroINorBHKKn8vcYwvb0kZsgPMoJZU8alDITGaWlkAYMOhFmvc4XcNrSCcMWUrfrI7YiULlt1/DljKceJwrNuDrUDQOr9GijmxoH+D9KoFyDiyUPpJxGQS2xf
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR01MB6780
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,
        SPF_HELO_PASS,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

1.) Background.
   1.1) In arm64, start a guest with Qemu which is running as a VMM of KVM,
        and bind the guest to core 33 and run program "a" in guest.
        The code of "a" shows below:
   	----------------------------------------------------------
		#include <stdio.h>

		int main()
		{
			unsigned long i = 0;

			for (;;) {
				i++;
			}

			printf("i:%ld\n", i);
			return 0;
		}
   	----------------------------------------------------------

   1.2) Use the following perf command in host:
      #perf stat -e cycles:G,cycles:H -C 33 -I 1000 sleep 1
          #           time             counts unit events
               1.000817400      3,299,471,572      cycles:G
               1.000817400          3,240,586      cycles:H

       This result is correct, my cpu's frequency is 3.3G.

   1.3) Use the following perf command in host:
      #perf stat -e cycles:G,cycles:H -C 33 -d -d  -I 1000 sleep 1
            time             counts unit events
     1.000831480        153,634,097      cycles:G                                                                (70.03%)
     1.000831480      3,147,940,599      cycles:H                                                                (70.03%)
     1.000831480      1,143,598,527      L1-dcache-loads                                                         (70.03%)
     1.000831480              9,986      L1-dcache-load-misses            #    0.00% of all L1-dcache accesses   (70.03%)
     1.000831480    <not supported>      LLC-loads
     1.000831480    <not supported>      LLC-load-misses
     1.000831480        580,887,696      L1-icache-loads                                                         (70.03%)
     1.000831480             77,855      L1-icache-load-misses            #    0.01% of all L1-icache accesses   (70.03%)
     1.000831480      6,112,224,612      dTLB-loads                                                              (70.03%)
     1.000831480             16,222      dTLB-load-misses                 #    0.00% of all dTLB cache accesses  (69.94%)
     1.000831480        590,015,996      iTLB-loads                                                              (59.95%)
     1.000831480                505      iTLB-load-misses                 #    0.00% of all iTLB cache accesses  (59.95%)

       This result is wrong. The "cycle:G" should be nearly 3.3G.

2.) Root cause.
	There is only 7 counters in my arm64 platform:
	  (one cycle counter) + (6 normal counters)

	In 1.3 above, we will use 10 event counters.
	Since we only have 7 counters, the perf core will trigger
       	multiplexing in hrtimer:
	     perf_mux_hrtimer_restart() --> perf_rotate_context().

       If the hrtimer occurs when the host is running, it's fine.
       If the hrtimer occurs when the guest is running,
       the perf_rotate_context() will program the PMU with filters for
       host context. The KVM does not have a chance to restore
       PMU registers with kvm_vcpu_pmu_restore_guest().
       The PMU does not work correctly, so we got wrong result.

3.) About this patch.
	Make a KVM_REQ_RELOAD_PMU request before reentering the
	guest. The request will call kvm_vcpu_pmu_restore_guest()
	to reconfigurate the filters for guest context.

4.) Test result of this patch:
      #perf stat -e cycles:G,cycles:H -C 33 -d -d  -I 1000 sleep 1
            time             counts unit events
     1.001006400      3,298,348,656      cycles:G                                                                (70.03%)
     1.001006400          3,144,532      cycles:H                                                                (70.03%)
     1.001006400            941,149      L1-dcache-loads                                                         (70.03%)
     1.001006400             17,937      L1-dcache-load-misses            #    1.91% of all L1-dcache accesses   (70.03%)
     1.001006400    <not supported>      LLC-loads
     1.001006400    <not supported>      LLC-load-misses
     1.001006400          1,101,889      L1-icache-loads                                                         (70.03%)
     1.001006400            121,638      L1-icache-load-misses            #   11.04% of all L1-icache accesses   (70.03%)
     1.001006400          1,031,228      dTLB-loads                                                              (70.03%)
     1.001006400             26,952      dTLB-load-misses                 #    2.61% of all dTLB cache accesses  (69.93%)
     1.001006400          1,030,678      iTLB-loads                                                              (59.94%)
     1.001006400                338      iTLB-load-misses                 #    0.03% of all iTLB cache accesses  (59.94%)

    The result is correct. The "cycle:G" is nearly 3.3G now.

Signed-off-by: Huang Shijie <shijie@os.amperecomputing.com>
---
v1 --> v2:
	Do not change perf/core code, only change the ARM64 kvm code.
	v1: https://lkml.org/lkml/2023/8/8/1465

---
 arch/arm64/kvm/arm.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index c2c14059f6a8..475a2f0e0e40 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -919,8 +919,17 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 		if (!ret)
 			ret = 1;
 
-		if (ret > 0)
+		if (ret > 0) {
+			/*
+			 * The perf_rotate_context() may rotate the events and
+			 * reprogram PMU with filters for host context.
+			 * So make a request before reentering the guest to
+			 * reconfigurate the event filters for guest context.
+			 */
+			kvm_make_request(KVM_REQ_RELOAD_PMU, vcpu);
+
 			ret = check_vcpu_requests(vcpu);
+		}
 
 		/*
 		 * Preparing the interrupts to be injected also
-- 
2.39.2