Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37;
Message-ID: <03b7da03-78a1-95b1-3969-634b5c9a5a56@amd.com>
Date:   Mon, 9 Oct 2023 22:33:41 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.12.0
Subject: Re: [Patch v4 07/13] perf/x86: Add constraint for guest perf metrics
 event
To:     Sean Christopherson <seanjc@google.com>,
        Peter Zijlstra <peterz@infradead.org>
Cc:     Ingo Molnar <mingo@kernel.org>,
        Dapeng Mi <dapeng1.mi@linux.intel.com>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Arnaldo Carvalho de Melo <acme@kernel.org>,
        Kan Liang <kan.liang@linux.intel.com>,
        Like Xu <likexu@tencent.com>,
        Mark Rutland <mark.rutland@arm.com>,
        Alexander Shishkin <alexander.shishkin@linux.intel.com>,
        Jiri Olsa <jolsa@kernel.org>,
        Namhyung Kim <namhyung@kernel.org>,
        Ian Rogers <irogers@google.com>,
        Adrian Hunter <adrian.hunter@intel.com>, kvm@vger.kernel.org,
        linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org,
        Zhenyu Wang <zhenyuw@linux.intel.com>,
        Zhang Xiong <xiong.y.zhang@intel.com>,
        Lv Zhiyuan <zhiyuan.lv@intel.com>,
        Yang Weijiang <weijiang.yang@intel.com>,
        Dapeng Mi <dapeng1.mi@intel.com>,
        David Dunn <daviddunn@google.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Mingwei Zhang <mizhang@google.com>,
        Jim Mattson <jmattson@google.com>,
        Like Xu <like.xu.linux@gmail.com>
References: <20231002115718.GB13957@noisy.programming.kicks-ass.net>
 <ZRrF38RGllA04R8o@gmail.com> <ZRroQg6flyGBtZTG@google.com>
 <20231002204017.GB27267@noisy.programming.kicks-ass.net>
 <ZRtmvLJFGfjcusQW@google.com>
 <20231003081616.GE27267@noisy.programming.kicks-ass.net>
 <ZRwx7gcY7x1x3a5y@google.com>
 <20231004112152.GA5947@noisy.programming.kicks-ass.net>
 <CAL715W+RgX2JfeRsenNoU4TuTWwLS5H=P+vrZK_GQVQmMkyraw@mail.gmail.com>
 <ZR3eNtP5IVAHeFNC@google.com> <ZR3hx9s1yJBR0WRJ@google.com>
 <c69a1eb1-e07a-8270-ca63-54949ded433d@gmail.com>
From:   Manali Shukla <manali.shukla@amd.com>
In-Reply-To: <c69a1eb1-e07a-8270-ca63-54949ded433d@gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QU9XdFVWanEvNVR2TEVMMTAzRkhSWVBYVmRrcUpheWtmVklXUTRlSldBY2Vq?=
 =?utf-8?B?R1FsTmxUNzNOSjA2dFl6K2dMcjg3N0Z6Nk1pSzcxV1laUTZURnVNNjNGU2U3?=
 =?utf-8?B?UjRBanVnamcwUE1IcEcvWWxqeFNwV3QrNG02OWV3cDRNN1lxSEFGMGZMTzZK?=
 =?utf-8?B?cDVacUpiM0RUaUxUVnMyejlCV3hHTkoxb0NMc1d1TXl5Q0FrSk15S3ZVblE3?=
 =?utf-8?B?T3hVWGwzYmJOZlR1Q1R3UFg3Y29COHZYUTNiTGdtTjZrMklUZnplRWpyK2VE?=
 =?utf-8?B?SXNEWVdKTmM5emxtNTYxUmNHbVNxMnQwSStrdmwxQlF3ZkZFTFRZOTJscndk?=
 =?utf-8?B?WDA0Vm8zUVdLblBQN0thcStkbDE2Zm5UU2QyT1BGcC81RE1SVHJyUm9RWHN2?=
 =?utf-8?B?ZlJ5TndXMnZ4RjRhUkwwTlhkUWJibHB2K1FtMnpNcFkzSkR1Z1Bib2YyMHBo?=
 =?utf-8?B?THVBcUpBWnk1TXBYemt0YVc5d09RalEwTUlBcXk2TFN6RFFRY2RGRnJmL0lH?=
 =?utf-8?B?UHVzdnk1Zk5UaEVLSTBrTERja0RabkZ0K1E4Z0tpUEVHanNoQnZ1NUFobktQ?=
 =?utf-8?B?L3Zua0haUG50N1BNS0JFUEZwd1hHRkZWQUIrVnV4QlBYeTUxZWlneVJBd2FG?=
 =?utf-8?B?MVp0L1o0cUk5QmJBWmhiMHltdWRKdlpKeUNHRjJJMzlSaG5OeURRNjBOTGs5?=
 =?utf-8?B?UXpnNUV4dzVFMUkzUnQza3llbjZONXVSVUQzOXFVVHdrT1hVNnhpbjZjVTJt?=
 =?utf-8?B?SnZxdnFhKyt4a29VVFZaWGpBSVhpL3BvaFBLYzhJdG9SMFFJdGZZeHBCMUVq?=
 =?utf-8?B?UjE1VmxsalNHL3BxWEJGeW14L21xd0pwM2RubXJ0bHlLSzBVbjlwcHd1YmdM?=
 =?utf-8?B?OEUvckNxN2Ewb1JPaGUxazlOTjRTOUdRb3p3Y0tYVWRKZXpTcmZyVjhONUVw?=
 =?utf-8?B?ek9VRHJtTndhb05KZVNnY2pXOXVub0drdHJqRWtQYjdoOENtdzUvcktDSXJF?=
 =?utf-8?B?MWN3R3l3RWk4SVMyL3Ywd3lLbzZpYkZCOTY3UktsamViT3dhbkdTNmpQN2pJ?=
 =?utf-8?B?Z1Zad1huNWpRUzYxZGJkUWtxWUxBb3FlbWFJTkNLT2RlWStDS0Rkc0ZyQnkr?=
 =?utf-8?B?K1hOTGZxQmxWSld6Zm8wajBYQnphVnUzSEtKTjZpNVQrTmh4OWlmSmwyQ1ZJ?=
 =?utf-8?B?Q3NkVHY4bEtiWGppUmFEcWx6RW1XalkwenE4MDZFYlFQVkx2Z1dkdVNFU3Er?=
 =?utf-8?B?R0tLZytaUHM5L1FrN1kxTlczTmUzL0lHbUVXT1BzUXRlMWpMUmZWSHFzdU56?=
 =?utf-8?B?ZGhERGVTdWk3eHVrR0pzRnJWSzZXbFZXS3o5d0xVSXR1THRtU0s4QlduVmg1?=
 =?utf-8?B?Si9pVTRsUWhRYkJOWVV5ZUl3alF0K3MxL2JoRCs4akVDUm1BbU9mUWl0R3pE?=
 =?utf-8?B?ck9uU0pOMXdyT3ZKSVVvamkrZU9zUHozcDdLVnNxUTAxL2xWN3gwMXp3M2VW?=
 =?utf-8?B?ampROW1LMncyb1V6ZWRUT215MVp3VDFZMi9aN1A5TnJpcFU3YUxNSTJnRjRT?=
 =?utf-8?B?ZTNXRDJzOUdyaTNMYjEydHBEQ0h3d0t3SGxocGRWOVc2LzFaeUg1cHo4UDRk?=
 =?utf-8?B?VE9pSWlaZlVmTURyZ3U0dEN1U3RyZytIOXZ3NnUrL3Q0bGtZMEs1OVozS3Rz?=
 =?utf-8?B?T1lRcEMrSlEwSGcwSkZPNW1sTUYxODByUWNTRnNWU2gxV3Z1S05meU91OUQ3?=
 =?utf-8?B?aHdBdDNRQjF5MllLZ0xsK282ZUwySkN3c2ZMeXB3UTJDSkJmN2I3ZXdjeGpJ?=
 =?utf-8?B?Mm43NEJTODNRZVZKL0FQMnV0S1ltYTlybHRWZ3UyWkQwdFVKaUc3MnVTaTFQ?=
 =?utf-8?B?V2gwTXZydlEyNUZIMDQ0Q0owTnRnM0g4VkhFWEVFNFROb1pWYnNlZVkvRlpK?=
 =?utf-8?B?dGtnTzlVZzNuQms3YzlFRWU5S29TczloMXVYcE1ScjNRbytqeC9Fb1dMOUhS?=
 =?utf-8?B?MVd5SE5aZ2RWQUVxSURaV2ZKdDkvblV2c3RtWmhQNGI5czlSZXVvSjVzVTV0?=
 =?utf-8?B?ZTNCUGZjWHc3Ylc2MUxxUDdmRTdzL2o1YjhmazdhWTl5R0liWkRlQnNTbUpl?=
 =?utf-8?Q?BDCEnJkx4iGx5aRxvbnKbG7LA?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9aa2bd0d-8195-4940-0515-08dbc8e9bb0e
X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB6214.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Oct 2023 17:03:59.6355
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: gYhO99rq4RjncTF/tA1zxS5/vgfrvSK/SP325gXLEzAvkOZqM+Be9YYX7TcHSLz0sq+evqy6tWs7qlXqjJQoLg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB6913
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        NICE_REPLY_A,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,
        SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 09 Oct 2023 10:04:14 -0700 (PDT)

On 10/8/2023 3:34 PM, Like Xu wrote:
> Hi all,
> 
> On 5/10/2023 6:05 am, Sean Christopherson wrote:
>> So I'll add a self-NAK to the idea of completely disabling the host PMU, I think
>> that would burn us quite badly at some point.
> 
> I seem to have missed a party, so allow me to add a few more comments
> to better facilitate future discussions in this direction:
> 
> (1) PMU counters on TEE
> 
> The SGX/SEV is already part of the upstream, but what kind of performance
> data will be obtained by sampling enclaves or sev-guest with hardware pmu
> counters on host (will the perf-report show these data missing holes or
> pure encrypted data), we don't have a clear idea nor have we established
> the right expectations. But on AMD profiling a SEV-SNP guest is supported:
> 
> "Fingerprinting attack protection is also not supported in the current
> generation of these technologies. Fingerprinting attacks attempt to
> determine what code the VM is running by monitoring its access patterns,
> performance counter information, etc." (AMD SEV-SNP White Paper, 2020)
> 
> (2) PMU Guest/Host Co-existence Development
> 
> The introduction of pt_mode in the KVM was misleading, leading subsequent
> developers to believe that static slicing of pmu facility usage was allowed.
> 
> On user scenarios, the host/perf should treat pmu resource requests from
> vCPUs with regularity (which can be unequal under the host's authority IMO)
> while allowing the host to be able to profile any software entity (including
> hypervisor and guest-code, including TEE code in debug mode). Functionality
> takes precedence over performance.
> 
> The semantics of exclude_guest/host should be tied to the hw-event isolation
> settings on the hardware interfaces, not to the human-defined sw-context.
> The perf subsystem is the arbiter of pmu resource allocation on the host,
> and any attempt to change the status quo (or maintenance scope) will not
> succeed. Therefore, vPMU developers are required to be familiar with the
> implementation details of both perf and kvm, and try not to add perf APIs
> dedicated to serving KVM blindly.
> 
> Getting host and guests to share limited PMU resources harmoniously is not
> particularly difficult compared to real rocket science in the kernel, so
> please don't be intimidated.
> 
> (3) Performance Concern in Co-existence
> 
> I wonder if it would be possible to add a knob to turn off the perf counter
> multiplexing mechanism on the host, so that in coexistence scenarios, the
> number of VM exits on the vCPU would not be increased by counter rotations
> due to timer expiration.
> 
> For normal counters shared between guest and host, the number of counter
> msr switches requiring a vm-entry level will be relatively small.
> (The number of counters is growing; for LBR, it is possible to share LBR
> select values to avoid frequent switching, but of course this requires the
> implementation of a software filtering mechanism when the host/guest read
> the LBR records, and some additional PMI; for DS-based PEBS, host and guest
> PEBS buffers are automatically segregated based on linear address).
> 
> There is a lot of room for optimisation here, and in real scenarios where
> triggering a large number of register switches in the host/guest PMU is
> to be expected and observed easily (accompanied by a large number of pmi
> appearances).
> 
> If we are really worried about the virtualisation overhead of vPMU, then
> virtio-pmu might be an option. In this technology direction, the back-end
> pmu can add more performance events of interest to the VM (including host
> un-core and off-core events, host-side software events, etc.) In terms of
> implementation, the semantics of the MSRLIST instruction can be re-used,
> along with compatibility with the different PMU hardware interfaces on ARM
> and Risc-v, which is also very friendly to production environments based on
> its virtio nature.
> 
> (4) New vPMU Feature Development
> 
> We should not put KVM's current vPMU support into maintenance-only mode.
> Users want more PMU features in the guest, like AMD vIBS, Intel pmu higher
> versions, Intel topdown and Arch lbr, more on the way. The maturity of
> different features' patch sets aren't the same, but we can't ignore these
> real needs because of available time for key maintainers, apathy towards
> contributors, mindset avoidance and laziness, and preference for certain
> technology stacks. These technical challenges will attract an influx of
> open source heroes to push the technology forward, which is good in the
> long run.
> 
> (5) More to think about
> 
> Similar to the guest PMU feature, the debugging feature may face the same
> state. For example, what happens when you debug code inside the host and
> guest at the same time (host debugs hypevisor/guest code and guest debugs
> guest code only) ?
> 
> Forgive my ignorance and offence, but we don't want to see a KVM subsystem
> controlled and driven by Google's demands.
> 
> Please feel free to share comments to move forward.
> 
> Thanks,
> Like Xu

Hi all,

I would like to add following things to the discussion just for the awareness of
everyone.

Fully virtualized PMC support is coming to an upcoming AMD SoC and we are
working on prototyping it.

As part of virtualized PMC design, the PERF_CTL registers are defined as Swap
type C: guest PMC states are loaded at VMRUN automatically but host PMC states
are not saved by hardware. If hypervisor is using the performance counters, it
is hypervisor's responsibility to save PERF_CTL registers to host save area
prior to VMRUN and restore them after VMEXIT. In order to tackle PMC overflow
interrupts in guest itself, NMI virtualization or AVIC can be used, so that
interrupt on PMC overflow in guest will not leak to host.

- Manali