Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Message-ID: <810c3148-1791-de57-27c0-d1ac5ed35fb8@gmail.com>
Date:   Wed, 16 Feb 2022 11:33:32 +0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.6.0
Subject: Re: KVM: x86: Reconsider the current approach of vPMU
Content-Language: en-US
To:     Sean Christopherson <seanjc@google.com>
Cc:     David Dunn <daviddunn@google.com>,
        Jim Mattson <jmattson@google.com>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Vitaly Kuznetsov <vkuznets@redhat.com>,
        Wanpeng Li <wanpengli@tencent.com>,
        Joerg Roedel <joro@8bytes.org>, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org,
        Stephane Eranian <eranian@google.com>,
        Peter Zijlstra <peterz@infradead.org>
References: <20220117085307.93030-1-likexu@tencent.com>
 <20220117085307.93030-3-likexu@tencent.com>
 <20220202144308.GB20638@worktop.programming.kicks-ass.net>
 <CALMp9eRBOmwz=mspp0m5Q093K3rMUeAsF3vEL39MGV5Br9wEQQ@mail.gmail.com>
 <2db2ebbe-e552-b974-fc77-870d958465ba@gmail.com>
 <YgPCm1WIt9dHuoEo@hirez.programming.kicks-ass.net>
 <YgQrWHCNG/s4EWFf@google.com>
 <39b64c56-bc8d-272d-da92-5aa29e54cdaf@gmail.com>
 <YgVHawnQWuSAk+C1@google.com>
From:   Like Xu <like.xu.linux@gmail.com>
Organization: Tencent
In-Reply-To: <YgVHawnQWuSAk+C1@google.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 11/2/2022 1:12 am, Sean Christopherson wrote:
> On Thu, Feb 10, 2022, Like Xu wrote:
>> On 10/2/2022 5:00 am, Sean Christopherson wrote:
>>> On Wed, Feb 09, 2022, Peter Zijlstra wrote:
>>>> Guests must not unilaterally steal the PMU.
>>>
>>> The proposal is to add an option to allow userspace to gift the PMU to the guest,
>>
>> Please define the verb "gift" in more details.
> 
> Add a knob that allows host userspace to control toggle between host perf having
> sole ownership of the PMU, versus ownership of the PMU being "gifted" to KVM guests
> upon VM-Entry and returned back to the host at VM-Exit.

For the vm-entry/exit level of granularity, we're able to do it without the host 
perf knob.
For the guest power-on/off level of granularity, perf does not compromise with KVM.

> 
> IIUC, it's the same idea as PT's PT_MODE_HOST_GUEST mode, just applied to the PMU.

TBH, I don't like the design of PT_MODE_HOST_GUEST, it breaks the flexibility.
I would prefer to see a transition in the use of PT to the existing vPMU approach.

> 
> By default, the host would have sole ownership, and access to the knob would be
> restricted appropriately.  KVM would disallow creation any VM that requires
> joint ownership, e.g. launching a TDX guest would require the knob to be enabled.

The knob implies a per-pCPU control granularity (internal or explicit), for 
scalability.

But again, regardless of whether the (TDX) guest has pmu enabled or not, the host
needs to use pmu (to profile host, non-TDX guest) without giving it away easily
at runtime (via knob).

We should not destroy the kind of hybrid usage, but going in a legacy direction,
complementing the lack of guest pmu functionality with emulation or
full context switching per the vm-entry/exit level of granularity.

> 
>> How do we balance the performance data collection needs of the
>> 'hypervisor user space' and the 'system-wide profiler user space' ?
> 
> If host userspace enables the knob and transitions into a joint ownership mode,
> then host userspace is explicitly acknowledging that it will no longer be able
> to profile KVM guests.

AFAI, most cloud provider don't want to lose this flexibility as it leaves
hundreds of "profile KVM guests" cases with nowhere to land.

> 
> Balancing between host and guest then gets factored into VM placement, e.g. VMs
> that need or are paying for access to the PMU can only land on systems that are
> configured for joint ownership.  If profiling the guest from the host is important,
> then place those guests only on hosts with sole ownership.

If the host user space does not reclaim PMU (triumph through prioritization) 
from the
guest (controlling this behavior is like controlling VMs placement), then the 
guest's
PMU functionality has nothing to lose, which is complete.