Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8;
From:   Tianyi Liu <i.pear@outlook.com>
To:     maz@kernel.org
Cc:     acme@kernel.org, adrian.hunter@intel.com,
        alexander.shishkin@linux.intel.com, i.pear@outlook.com,
        irogers@google.com, jolsa@kernel.org, kvm@vger.kernel.org,
        kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org,
        linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org,
        mark.rutland@arm.com, mingo@redhat.com, namhyung@kernel.org,
        pbonzini@redhat.com, peterz@infradead.org, seanjc@google.com,
        x86@kernel.org
Subject: Re: [PATCH v2 0/5] perf: KVM: Enable callchains for guests
Date:   Thu, 12 Oct 2023 14:35:42 +0800
Message-ID: <SY4P282MB108434CA47F2C55E490A1C6B9DD3A@SY4P282MB1084.AUSP282.PROD.OUTLOOK.COM>
In-Reply-To: <8734yhm7km.wl-maz@kernel.org>
References: <8734yhm7km.wl-maz@kernel.org>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-MessageSentRepresentingType: 1
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: SY4P282MB1084:EE_|ME0P282MB4566:EE_
X-MS-Office365-Filtering-Correlation-Id: 6d30b3c1-a455-4072-7769-08dbcaed9fe5
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: EnwI2d6YhylB4yVn9GiD8ne6Wq1WbWkRHv9Vn/JqujEx43bXRo+s3x5n/rCwHKx6hB43Dq7LY+G8l8G0JDLx928m0v2km0/BHUa44hnH/J1X0DeTCIEUe/6l0rzLepP73mkajOl9Vj8iSCMbX3XvR+9LBcbuJyzbjTlixMQQaE9jow3UtCNEEZ6UEY/vChSgT1AWVZLVoNjYuy6COzC9JuejxAK/iSYqZt0wcKFM45y3lx3yuH+L4nTC9bdB35CoXieuxkP22Jpr3AHfZ3vOk0V4LwUKysW3YUmSaPfOxdEMhWzVFNo76oPMifNshEed4OB/+hw809sXLCgVlaBvrgyfnzOmmxfedAcrLSp6L5O0+VYS87CKnVodD5i1fLzVLVXkhHAiBe2uRtKy5kkxLPJar0ww2gpGNFk7zLf2Z4uobvxxfl82hIwYLcTSM1NO0t+zYwB2L4m0zXV4YSKGBXjrq6EB2YuPS9W5atar6pryCThHcVNCbkHeE30qW0GHkUkQMnwSpyFNtDnUQSgXbrs1FGKGVCCOjAGqKml0VGjHDS0HXcH5zNvZTrtUOVcNv6m6PdGQVPa29vb0kJuTYD+XSVubG5CK8paXVz6pOP2yWEN+K7yLSzckySwwE3h3
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?SZlwvVk9U/VxBRpNav8g8J9W6y4cCoQ7sMRQPNOjMivvAUrT6L1YQ2i/7Bee?=
 =?us-ascii?Q?MIlX2ZGNELi2l00p/LAlA1IkiC0JHuN6jsfXTwP/ZIOatOlDPqDsh/9URUyU?=
 =?us-ascii?Q?7jqAKecjQciu6XY2MQQxQe6nSJA/dBjfbAo8NPu8Cwguy7T7fCTCkWelFCf4?=
 =?us-ascii?Q?3Xcy0CH3DSFqhzziVeXZXkUfcnpEDM7ww8s5jUnuu2fYbRji4NFUUu5FFgvb?=
 =?us-ascii?Q?nk0vDAAuzWMVumneEv4SnhHnSrpMXgdkMlSAHjAfUi15MuaVr8cCWMR16sp0?=
 =?us-ascii?Q?emKhq6ln4xV65sUmjYnZeCYTb2G8f6EMZwwxRt5PMdHS8k2J0zonQ/eJFk9U?=
 =?us-ascii?Q?azX3WkokAZJVe5r26SEH8JgsCiiGGFwui0ZIk03JY7eNKWEN0qZEiJQdCxMZ?=
 =?us-ascii?Q?DumRX3cDnaAgBpsjhJJpWRMgEwrbu+VHvG0ZsdlHzji5C15WLS85u+PtFI5o?=
 =?us-ascii?Q?NeNJpVYPceipBMc4ru7RQ+5leVGjGZJXKYbkmJ08wGg9oLkQtwlei01Wwuvu?=
 =?us-ascii?Q?uv58VQsFDbYr8al3ytXWvPRTtoc6hsg16gtpFnog12afYnEnC6ZM4p57zDlq?=
 =?us-ascii?Q?wdCjmK2uygTrbCl+MYw/8joniyNJBMLRhotHVJpcsNfvk4uPHLsGrYOWdNxJ?=
 =?us-ascii?Q?P5hBzG+5tcPuhR2DQ6CCdIPaslWhhbhSJ0ZlA6UimqF4sSMyn8kw/whLwGUv?=
 =?us-ascii?Q?Y9vMxyLd2FuOJPue8f26uSdhBLGPe7+3Q/8K8Z4RMVX/Iy1GOjRr8lpv5kyP?=
 =?us-ascii?Q?JCIqxWSBtDCcDmyEOOOytHuxYtWMI+beyX28hpdvAAWKMlj8wMIGepwTXKNF?=
 =?us-ascii?Q?syX4Xc5muq0VU3I/pmdN4yv3/KBMaSOWZKd6vyL+oUPmYzFhTe3VAA+37K8Z?=
 =?us-ascii?Q?bS2SsU49ccxyc6WKy+w7BbcsmnycSNR6AE7KjQZ/I2zCCAncAU9H1Rn1bCS/?=
 =?us-ascii?Q?MuyxiNX+c5kGV4+nziP76PhrlgB9A2y6ER5xfxE2oeola2PLm9WVCa4H9iMd?=
 =?us-ascii?Q?vcQT1LOYrlVN6ILYhur/b2px+LulJMFCmrt13hGN4BnqsDfP3sBokBsXSyO9?=
 =?us-ascii?Q?KoyoPnyTXd99LHahg/b6m0etABIYuxiU/hVZvi7UNBEgMssXS+GuPYbr2C9z?=
 =?us-ascii?Q?CNqKr8teiwiKFk0rG5GHFF+ldPqIc7wnJ+sP8Wyi5qJJrZCaRTxNRgTR+WiA?=
 =?us-ascii?Q?VZrsbCgwwcvoBr7DfoWJpuQY+W/75Cu7ctvV+zFmnYFqM3nGMkfG3GK1NR9b?=
 =?us-ascii?Q?CSiNMgJBP5JexrDOKUdR?=
X-OriginatorOrg: outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6d30b3c1-a455-4072-7769-08dbcaed9fe5
X-MS-Exchange-CrossTenant-AuthSource: SY4P282MB1084.AUSP282.PROD.OUTLOOK.COM
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Oct 2023 06:36:55.4181
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000
X-MS-Exchange-Transport-CrossTenantHeadersStamped: ME0P282MB4566
X-Spam-Status: No, score=-0.6 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,
	HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,
	SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Wed, 11 Oct 2023 23:37:18 -0700 (PDT)

Hi Marc,

On Sun, 11 Oct 2023 16:45:17 +0000, Marc Zyngier wrote:
> > The event processing flow is as follows (shown as backtrace):
> >   #0 kvm_arch_vcpu_get_frame_pointer / kvm_arch_vcpu_read_virt (per arch)
> >   #1 kvm_guest_get_frame_pointer / kvm_guest_read_virt
> >      <callback function pointers in `struct perf_guest_info_callbacks`>
> >   #2 perf_guest_get_frame_pointer / perf_guest_read_virt
> >   #3 perf_callchain_guest
> >   #4 get_perf_callchain
> >   #5 perf_callchain
> >
> > Between #0 and #1 is the interface between KVM and the arch-specific
> > impl, while between #1 and #2 is the interface between Perf and KVM.
> > The 1st patch implements #0. The 2nd patch extends interfaces between #1
> > and #2, while the 3rd patch implements #1. The 4th patch implements #3
> > and modifies #4 #5. The last patch is for userspace utils.
> >
> > Since arm64 hasn't provided some foundational infrastructure (interface
> > for reading from a virtual address of guest), the arm64 implementation
> > is stubbed for now because it's a bit complex, and will be implemented
> > later.
> 
> I hope you realise that such an "interface" would be, by definition,
> fragile and very likely to break in a subtle way. The only existing
> case where we walk the guest's page tables is for NV, and even that is
> extremely fragile.
>

For walking the guest's page tables, yes, there're only very few
use cases. Most of them are used in nested virtualization and XEN.

> Given that, I really wonder why this needs to happen in the kernel.
> Userspace has all the required information to interrupt a vcpu and
> walk its current context, without any additional kernel support. What
> are the bits here that cannot be implemented anywhere else?
>

Thanks for pointing this out, I agree with your opinion.
Whether it's walking guest's contexts or performing an unwind,
user space can indeed accomplish these tasks.
The only reasons I see for implementing them in the kernel are performance
and the access to a broader range of PMU events.

Consider if I were to implement these functionalities in userspace:
I could have `perf kvm` periodically access the guest through the KVM API
to retrieve the necessary information. However, interrupting a VCPU
through the KVM API from user space might introduce higher latency
(not tested specifically), and the overhead of syscalls could also
limit the sampling frequency.

Additionally, it seems that user space can only interrupt the VCPU
at a certain frequency, without harnessing the richness of the PMU's
performance events. And if we incorporate the logic into the kernel,
`perf kvm` can bind to various PMU events and sample with a faster
performance in PMU interrupts.

So, it appears to be a tradeoff -- whether it's necessary to introduce
more complexity in the kernel to gain access to a broader range and more
precise performance data with less overhead. In my current use case,
I just require simple periodic sampling, which is sufficient for me,
so I'm open to both approaches.

> > Tianyi Liu (5):
> >   KVM: Add arch specific interfaces for sampling guest callchains
> >   perf kvm: Introduce guest interfaces for sampling callchains
> >   KVM: implement new perf interfaces
> >   perf kvm: Support sampling guest callchains
> >   perf tools: Support PERF_CONTEXT_GUEST_* flags
> >
> >  arch/arm64/kvm/arm.c                | 17 +++++++++
> 
> Given that there is more to KVM than just arm64 and x86, I suggest
> that you move the lack of support for this feature into the main KVM
> code.

Currently, sampling for KVM guests is only available for the guest's
instruction pointer, and even the support is limited, it is available
on only two architectures (x86 and arm64). This functionality relies on
a kernel configuration option called `CONFIG_GUEST_PERF_EVENTS`,
which will only be enabled on x86 and arm64.
Within the main KVM code, these interfaces are enclosed within
`#ifdef CONFIG_GUEST_PERF_EVENTS`. Do you think these are enough?

Best regards,
Tianyi Liu