Received: by 2002:a05:6a10:17d3:0:0:0:0 with SMTP id hz19csp434043pxb; Wed, 14 Apr 2021 20:25:02 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyAigr1y4HwNbnt9XBp5oAS1ZHWKP/NnqCqfMd3FYNwlAItHD51nxkacYxjMqSoKzdZOH/2 X-Received: by 2002:a17:906:4004:: with SMTP id v4mr1250391ejj.208.1618457102218; Wed, 14 Apr 2021 20:25:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1618457102; cv=none; d=google.com; s=arc-20160816; b=ZW2NlqrC+4p3UFXWznkODaU8Dv58StOsm9ftsMaolcDlzSTOOx/x8DrTsYTtJAoFtM lRrBtiajf5ex9QteKGaFmDtC1K/wlciI35I06ItQeEZnOcgecuxcCdNBN2EwKck9JBc7 Hi9X7sju+Ek968bED+bAQj4Of4pQSk4k3Pb7L10Uw3MndnBgDxr4LkY7ZMCSISbA0hyB FjRDiM3VdTyqUeeEO1fEj5CUzb8XTo1s8DGBpsYJKTEEryINaj8U7qid2fqKFNbGVPK+ Aq9xsAGXShwRTys8bkIy1BoE8xwTtBEEC2GV4OyO2n4TGz2M/Mt27kRxFVSOoRoJ4yu5 5IAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:content-language :in-reply-to:mime-version:user-agent:date:message-id:organization :from:references:cc:to:subject:ironport-sdr:ironport-sdr; bh=7vZMZRG4+UgaWNnfCCSCeV/fieB7DvSyMKVQ8FTBu6o=; b=QSCVGcpKZ0r99/PCNrbuPuRBTPqfCUq88DU7JK8ksTTDxq3hKI1wRfrGSdN1FEhko4 B9W0K1Z5BXKGD852pyd+JmELYyV7sP+/XG4/voE/mzJ+AIwVIjRUwhb+MdbrqW6lqPvT nAFvPgXR3iBd5Ayv6N5k1b9GMiIlB32b1oyy1XK+VApSNnvakIWW/XcotQL5Fb+dc7iy 4RlLUk5kBNUHANAy7baABXmdWfEVqhs5Xv05fwovAOzdbkU8mIfTiTA4cYl+i2Lkvhd8 fplDtGn1xW6hASiagAuezrgiqaOwRW/y1HrAyt2NMjpoYoq1bFRwg9N8XYUcoeuSnkhw 2F2Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id f7si1337676edd.239.2021.04.14.20.24.39; Wed, 14 Apr 2021 20:25:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229934AbhDODYI (ORCPT + 99 others); Wed, 14 Apr 2021 23:24:08 -0400 Received: from mga09.intel.com ([134.134.136.24]:48391 "EHLO mga09.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229853AbhDODYG (ORCPT ); Wed, 14 Apr 2021 23:24:06 -0400 IronPort-SDR: iDoUQyd2cXMvY3qodPNaVvUwslDQ8IoH9TaNlL2Bk2KZWObw+ZFN/oCYHS9Q0iGMO+J9EiIWBL St2vP/QvOXBA== X-IronPort-AV: E=McAfee;i="6200,9189,9954"; a="194889886" X-IronPort-AV: E=Sophos;i="5.82,223,1613462400"; d="scan'208";a="194889886" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Apr 2021 20:23:44 -0700 IronPort-SDR: nlm/WecwwNqgbtHsmzB33KM9/VNaHcu+Py3c0xb8JQFANHdm+fY67fvqqZ4G4Da6UYyf4t55S8 QjnQBbrKmAsg== X-IronPort-AV: E=Sophos;i="5.82,223,1613462400"; d="scan'208";a="425014595" Received: from likexu-mobl1.ccr.corp.intel.com (HELO [10.238.4.93]) ([10.238.4.93]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Apr 2021 20:23:41 -0700 Subject: Re: [PATCH v4 01/16] perf/x86/intel: Add x86_pmu.pebs_vmx for Ice Lake Servers To: Liuxiangdong Cc: andi@firstfloor.org, "Fangyi (Eric)" , Xiexiangyou , kan.liang@linux.intel.com, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, wei.w.wang@intel.com, x86@kernel.org, "Xu, Like" References: <20210329054137.120994-2-like.xu@linux.intel.com> <606BD46F.7050903@huawei.com> <18597e2b-3719-8d0d-9043-e9dbe39496a2@intel.com> <60701165.3060000@huawei.com> <1ba15937-ee3d-157a-e891-981fed8b414d@linux.intel.com> <607700F2.9080409@huawei.com> <76467c36-3399-a123-d582-92affadc4d73@intel.com> <6077A9AE.2090705@huawei.com> From: Like Xu Organization: Intel OTC Message-ID: Date: Thu, 15 Apr 2021 11:23:39 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.9.1 MIME-Version: 1.0 In-Reply-To: <6077A9AE.2090705@huawei.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2021/4/15 10:49, Liuxiangdong wrote: > > > On 2021/4/15 9:38, Xu, Like wrote: >> On 2021/4/14 22:49, Liuxiangdong wrote: >>> Hi Like, >>> >>> On 2021/4/9 16:46, Like Xu wrote: >>>> Hi Liuxiangdong, >>>> >>>> On 2021/4/9 16:33, Liuxiangdong (Aven, Cloud Infrastructure Service >>>> Product Dept.) wrote: >>>>> Do you have any comments or ideas about it ? >>>>> >>>>> https://lore.kernel.org/kvm/606E5EF6.2060402@huawei.com/ >>>> >>>> My expectation is that there may be many fewer PEBS samples >>>> on Skylake without any soft lockup. >>>> >>>> You may need to confirm the statement >>>> >>>> "All that matters is that the EPT pages don't get >>>> unmapped ever while PEBS is active" >>>> >>>> is true in the kernel level. >>>> >>>> Try "-overcommit mem-lock=on" for your qemu. >>>> >>> >>> Sorry, in fact, I don't quite understand >>> "My expectation is that there may be many fewer PEBS samples on Skylake >>> without any soft lockup. " >> >> For testcase: perf record -e instructions:pp ./workload >> >> We can get 2242 samples on the ICX guest, but >> only 17 samples or less on the Skylake guest. >> >> In my testcase on Skylake, neither the host nor the guest triggered the >> soft lock. >> > > Thanks for your explanation! > Could you please show your complete qemu command and qemu version used on > Skylake? > I hope I can test it again according to your qemu cmd and version. A new version is released and you may have a try. qemu command: "-enable-kvm -cpu host,migratable=no" qemu base commit: db55d2c9239d445cb7f1fa8ede8e42bd339058f4 kvm base commit: f96be2deac9bca3ef5a2b0b66b71fcef8bad586d diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c index 63c55f45ca92..727f55400eaf 100644 --- a/arch/x86/events/intel/core.c +++ b/arch/x86/events/intel/core.c @@ -5618,6 +5618,7 @@ __init int intel_pmu_init(void) case INTEL_FAM6_KABYLAKE: case INTEL_FAM6_COMETLAKE_L: case INTEL_FAM6_COMETLAKE: + x86_pmu.pebs_vmx = 1; x86_add_quirk(intel_pebs_isolation_quirk); x86_pmu.late_ack = true; memcpy(hw_cache_event_ids, skl_hw_cache_event_ids, sizeof(hw_cache_event_ids)); diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c index 100a749251b8..9e37e3dbe3ae 100644 --- a/arch/x86/kvm/pmu.c +++ b/arch/x86/kvm/pmu.c @@ -150,9 +150,8 @@ static void pmc_reprogram_counter(struct kvm_pmc *pmc, u32 type, * the accuracy of the PEBS profiling result, because the "event IP" * in the PEBS record is calibrated on the guest side. */ - attr.precise_ip = 1; - if (x86_match_cpu(vmx_icl_pebs_cpu) && pmc->idx == 32) - attr.precise_ip = 3; + attr.precise_ip = x86_match_cpu(vmx_icl_pebs_cpu) ? + ((pmc->idx == 32) ? 3 : 1) : ((pmc->idx == 1) ? 3 : 1); } event = perf_event_create_kernel_counter(&attr, -1, current, > > >>> >>> And, I have used "-overcommit mem-lock=on"  when soft lockup happens. >> >> I misunderstood the use of "mem-lock=on". It is not the same as the >> guest mem pin and I believe more kernel patches are needed. >> >>> >>> >>> Now, I have tried to configure 1G-hugepages for 2G-mem vm. Each of guest >>> numa nodes has 1G mem. >>> When I use pebs(perf record -e cycles:pp) in guest, there are successful >>> pebs samples just for a while and >>> then I cannot get pebs samples. Host doesn't soft lockup in this process. >> >> In the worst case, no samples are expected. >> >>> >>> Are there something wrong on skylake for we can only get a few samples? >>> IRQ?  Or using hugepage is not effecitve? >> >> The few samples comes from hardware limitation. >> The Skylake doesn't have this "EPT-Friendly PEBS" capabilityand >> some PEBS records will be lost when used by guests. >> >>> >>> Thanks! >>> >>>>> >>>>> >>>>> On 2021/4/6 13:14, Xu, Like wrote: >>>>>> Hi Xiangdong, >>>>>> >>>>>> On 2021/4/6 11:24, Liuxiangdong (Aven, Cloud Infrastructure Service >>>>>> Product Dept.) wrote: >>>>>>> Hi,like. >>>>>>> Some questions about this new pebs patches set: >>>>>>> https://lore.kernel.org/kvm/20210329054137.120994-2-like.xu@linux.intel.com/ >>>>>>> >>>>>>> >>>>>>> The new hardware facility supporting guest PEBS is only available >>>>>>> on Intel Ice Lake Server platforms for now. >>>>>> >>>>>> Yes, we have documented this "EPT-friendly PEBS" capability in the SDM >>>>>> 18.3.10.1 Processor Event Based Sampling (PEBS) Facility >>>>>> >>>>>> And again, this patch set doesn't officially support guest PEBS on >>>>>> the Skylake. >>>>>> >>>>>>> >>>>>>> >>>>>>> AFAIK, Icelake supports adaptive PEBS and extended PEBS which >>>>>>> Skylake doesn't. >>>>>>> But we can still use IA32_PEBS_ENABLE MSR to indicate >>>>>>> general-purpose counter in Skylake. >>>>>> >>>>>> For Skylake, only the PMC0-PMC3 are valid for PEBS and you may >>>>>> mask the other unsupported bits in the pmu->pebs_enable_mask. >>>>>> >>>>>>> Is there anything else that only Icelake supports in this patches set? >>>>>> >>>>>> The PDIR counter on the Ice Lake is the fixed counter 0 >>>>>> while the PDIR counter on the Sky Lake is the gp counter 1. >>>>>> >>>>>> You may also expose x86_pmu.pebs_vmx for Skylake in the 1st patch. >>>>>> >>>>>>> >>>>>>> >>>>>>> Besides, we have tried this patches set in Icelake.  We can use >>>>>>> pebs(eg: "perf record -e cycles:pp") >>>>>>> when guest is kernel-5.11, but can't when kernel-4.18. Is there a >>>>>>> minimum guest kernel version requirement? >>>>>> >>>>>> The Ice Lake CPU model has been added since v5.4. >>>>>> >>>>>> You may double check whether the stable tree(s) code has >>>>>> INTEL_FAM6_ICELAKE in the arch/x86/include/asm/intel-family.h. >>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Xiangdong Liu >>>>>> >>>>> >>>> >>> >> >