Received: by 2002:a05:6a10:206:0:0:0:0 with SMTP id 6csp509632pxj; Wed, 2 Jun 2021 04:58:55 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzu/kPUm+86Q2DHOYIqMCeAW7N6KSyDa++yaWX5RNGvZ6QkLDS7p0iIzqkNCNMqfmrahGIy X-Received: by 2002:a05:6402:685:: with SMTP id f5mr6219226edy.178.1622635134971; Wed, 02 Jun 2021 04:58:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1622635134; cv=none; d=google.com; s=arc-20160816; b=VkKpV+gle1iWv466fGkRhEUbZBiv52CAFV0nGnLbrEOfo5nnB2KIQwmTuXpj7Fv2NN qcWNFZtla3AQ4pWzu0I8TWORgtCbP/FJ2ECFDX171HqdvUdw/1dBA4v1C969XzeYPluy M2lLoiSwFvuCvUiBRcTrMI/lrwvpcM5+HllH/hVOR765H1eVRv9SZT840cRReARFpaVz e437lyZz0uMOu4zP5HTqvkBgb2z89jJJ6HRJRYOLA4kMN3NGer6Kf3QUdEv6naXHTvvq eoU2HDJ/HBeMufivX5vZ0hKOfcE+sVBqYCAOoCXIwpBREr7GkLhS6+X9Dp3S49oBjh9d ge7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=rnUVpWK+YZJQPEFOfjkkzAmsNNApKgBQ8SohcHgKJEg=; b=E1qUifdVwsqHROxMBNbjORY6yC63Hu9xS9SDbXz6xDHSnI1eFfk2hRwYoQ5G9r0ZKq GYlyAJ6/UmqkkgeJYiuV+/qB+xxu0yY7P4b/mro4ZnGbAUwlXB2xhoSmfxvPpEVPMALT 8yZk5DnWcNLlH7KckIUwO0Ye2InPZIw8qhi2eueLNM/2slHf/xGU8u28VHdtf06VE6aM I5g4om1omdZB0ZouUt+gIGCah0Hm68deAthc6pdRRtG8sonw37JK10lJwzuvDf7tyoVC V01wcclELq71fpC2706WsPhPj5OekOGfaplHe/ujf+hREA5bH/pJYrKd/T7PwOUAHC8N gRBw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=IIS0hSol; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id ec2si18733807ejb.189.2021.06.02.04.58.32; Wed, 02 Jun 2021 04:58:54 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=IIS0hSol; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229807AbhFBLy2 (ORCPT + 99 others); Wed, 2 Jun 2021 07:54:28 -0400 Received: from mail.kernel.org ([198.145.29.99]:51252 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229471AbhFBLy1 (ORCPT ); Wed, 2 Jun 2021 07:54:27 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 366ED61242; Wed, 2 Jun 2021 11:52:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1622634764; bh=I//4RBKnWt9wBavL0VfmU2oqhX+9WjJBJW64dSVgtEE=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=IIS0hSol21RczplZ6KQVFLpb+OTq1t+cRqvQ+ADWIp9yDbNpr3wJfKrmRGTzYuWKm ogcJflRhNKmcouIS7VMlOEXTf/pgrO7O47A28U7zFA0ILaDcFYNhMwSFC3TLOhz6cw trD4/NH9mz2mKqsAIjNOglrNPmbuGjUu3avHZhshQNdxvA+ACvKvptNNnWHKDR7de1 teMbZ35EI4zYgfoPIWdb8nsRbUdGDslkY64mz8marLR4nLQM7pm3jsIRIX1yLkcwf2 0Ax4u6V17QjovjZRky2FEsFHa5+zY06p8RpuKwjS/2cDdjhRva3p2Tk2r2dMcGjkz+ Cj6P6gRU8cWlw== Received: by quaco.ghostprotocols.net (Postfix, from userid 1000) id 1A44E4011C; Wed, 2 Jun 2021 08:52:41 -0300 (-03) Date: Wed, 2 Jun 2021 08:52:41 -0300 From: Arnaldo Carvalho de Melo To: Masami Hiramatsu Cc: Ravi Bangoria , Jiri Olsa , linux-kernel@vger.kernel.org, aneesh.kumar@linux.ibm.com, Peter Zijlstra , Ingo Molnar , Namhyung Kim , Ian Rogers Subject: Re: [PATCH] perf probe: Provide more detail with relocation warning Message-ID: References: <20210525043744.193297-1-ravi.bangoria@linux.ibm.com> <20210525214858.33a66846ac09e499c3268a63@kernel.org> <05e32c82-1009-03ba-d973-8b1bc0582ce2@linux.ibm.com> <20210526153340.a49ba8292f201493990f210c@kernel.org> <20210526232020.c1632c2285af811c7531b3cc@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20210526232020.c1632c2285af811c7531b3cc@kernel.org> X-Url: http://acmel.wordpress.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Em Wed, May 26, 2021 at 11:20:20PM +0900, Masami Hiramatsu escreveu: > On Wed, 26 May 2021 09:56:29 -0300 > Arnaldo Carvalho de Melo wrote: > > > Em Wed, May 26, 2021 at 03:33:40PM +0900, Masami Hiramatsu escreveu: > > > On Wed, 26 May 2021 10:23:18 +0530 Ravi Bangoria wrote: > > > > On 5/25/21 6:18 PM, Masami Hiramatsu wrote: > > > > > On Tue, 25 May 2021 10:07:44 +0530 Ravi Bangoria wrote: > > > > > > >> When run as normal user with default sysctl kernel.kptr_restrict=0 > > > > >> and kernel.perf_event_paranoid=2, perf probe fails with: > > > > > > >> $ ./perf probe move_page_tables > > > > >> Relocated base symbol is not found! > > > > > > >> The warning message is not much informative. The reason perf > > > > >> fails is because /proc/kallsyms is restricted by > > > > >> perf_event_paranoid=2 for normal user and thus perf fails to read > > > > >> relocated address of the base symbol. > > > > > > >> Tweaking kptr_restrict and perf_event_paranoid can change the > > > > >> behavior of perf probe. Also, running as root or privileged user > > > > >> works too. Add these details in the warning message. > > > > > > >> Plus, kmap->ref_reloc_sym might not be always set even if > > > > >> host_machine is initialized. Above is the example of the same. > > > > >> Remove that comment. > > > > > > > Yes, those are restricted in some cases. Anyway without priviledged > > > > > (super) user, perf probe can not set the probe in ftrace. > > > > > > > Hmm, I think it should check the effective user-id at first. If it > > > > > is not super user and the action will access tracefs and kallsyms, > > > > > it should warn at that point. > > > > > > If kptr_restrict=2, perf probe fails with same error even for root user. > > > > That's why I thought to just change this warning message. > > > > > Ah, yes. In that case, perf probe must not use the base symbol. > > > (like -D option) > > > OK, then, let's merge this fix. > > > > > Acked-by: Masami Hiramatsu > > > > Thanks, applied as it improves the current situation. > > > > But as a follow up, to further improve this, we can reuse what 'perf trace' has: > > > > $ perf trace sleep 1 > > Error: No permissions to read /sys/kernel/tracing/events/raw_syscalls/sys_(enter|exit) > > Hint: Try 'sudo mount -o remount,mode=755 /sys/kernel/tracing/' > > $ sudo mount -o remount,mode=755 /sys/kernel/tracing/ > > $ perf trace sleep 1 > > Error: Permission denied. > > Hint: Check /proc/sys/kernel/perf_event_paranoid setting. > > Hint: For your workloads it needs to be <= 1 > > Hint: For system wide tracing it needs to be set to -1. > > Hint: Try: 'sudo sh -c "echo -1 > /proc/sys/kernel/perf_event_paranoid"' > > Hint: The current value is 2. > > $ > > OK, let me check this. > BTW, does perf_event_paranoid affect only perf syscall (and kallsyms), > not the tracefs correct? > > > I.e. go the extra step and show what the current value is and what it > > needs to be to achieve what is being attempted. > > > > IOW combine error message with relevant documentation, to save steps. > > > > See what 'perf top' does for an unpriv user: > > > > $ perf top --stdio > > Error: > > Access to performance monitoring and observability operations is limited. > > Enforced MAC policy settings (SELinux) can limit access to performance > > monitoring and observability operations. Inspect system audit records for > > more perf_event access control information and adjusting the policy. > > Consider adjusting /proc/sys/kernel/perf_event_paranoid setting to open > > access to performance monitoring and observability operations for processes > > without CAP_PERFMON, CAP_SYS_PTRACE or CAP_SYS_ADMIN Linux capability. > > More information can be found at 'Perf events and tool security' document: > > https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html > > perf_event_paranoid setting is 2: > > -1: Allow use of (almost) all events by all users > > Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK > > >= 0: Disallow raw and ftrace function tracepoint access > > >= 1: Disallow CPU event access > > >= 2: Disallow kernel profiling > > To make the adjusted perf_event_paranoid setting permanent preserve it > > in /etc/sysctl.conf (e.g. kernel.perf_event_paranoid = ) > > Hmm, I would rather like pointing manpages... Man pages are long, if you quote the relevant part of it when the problem takes place, IMHO it helps the user. - Arnaldo > Would we better to have perf-security.7 manpage? > > Thank you, > > > $ > > > > - Arnaldo > > > > > > > > > > > > > Different combinations of privilege, perf_event_paranoid, kptr_restrict: > > > > > > > > Normal/Root user > > > > | perf_event_paranoid > > > > V V kptr_restrict perf probe error > > > > ---------------------------------------------------------------- > > > > N -1 0 Failed to open kprobe_events: Permission denied > > > > N 0 0 Failed to open kprobe_events: Permission denied > > > > N 1 0 Failed to open kprobe_events: Permission denied > > > > N 2 0 Relocated base symbol is not found! > > > > > > > > N -1 1 Relocated base symbol is not found! > > > > N 0 1 Relocated base symbol is not found! > > > > N 1 1 Relocated base symbol is not found! > > > > N 2 1 Relocated base symbol is not found! > > > > > > > > N -1 2 Relocated base symbol is not found! > > > > N 0 2 Relocated base symbol is not found! > > > > N 1 2 Relocated base symbol is not found! > > > > N 2 2 Relocated base symbol is not found! > > > > > > > > R -1 0 No error. > > > > R 0 0 No error. > > > > R 1 0 No error. > > > > R 2 0 No error. > > > > > > > > R -1 1 No error. > > > > R 0 1 No error. > > > > R 1 1 No error. > > > > R 2 1 No error. > > > > > > > > R -1 2 Relocated base symbol is not found! > > > > R 0 2 Relocated base symbol is not found! > > > > R 1 2 Relocated base symbol is not found! > > > > R 2 2 Relocated base symbol is not found! > > > > > > > > Ravi > > > > > > > > > -- > > > Masami Hiramatsu > > > > -- > > > > - Arnaldo > > > -- > Masami Hiramatsu -- - Arnaldo