Date:   Thu, 16 Mar 2023 14:18:57 +0800
From:   Yujie Liu <yujie.liu@intel.com>
To:     Adrian Hunter <adrian.hunter@intel.com>
CC:     <oe-lkp@lists.linux.dev>, <lkp@intel.com>,
        <linux-kernel@vger.kernel.org>,
        Arnaldo Carvalho de Melo <acme@redhat.com>,
        Namhyung Kim <namhyung@kernel.org>,
        Ian Rogers <irogers@google.com>, Jiri Olsa <jolsa@kernel.org>,
        <linux-perf-users@vger.kernel.org>
Subject: Re: [linus:master] [perf symbols] ce4c8e7966:
 perf-sanity-tests.Test_data_symbol.fail
Message-ID: <ZBK00bqcyl2lnG8W@yujie-X299>
References: <202303061424.6ad43294-yujie.liu@intel.com>
 <d62c6cc7-9f43-2c23-d9e2-1ddbd0f5234c@intel.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <d62c6cc7-9f43-2c23-d9e2-1ddbd0f5234c@intel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?8W8oX8jj0raZe1c75O6JTAChNiSe2hDX4EGeCildlvI/cccB8ZKoDVfshyNA?=
 =?us-ascii?Q?Q6s2pez8cUrRXD1MPTYoThEq41JJtTSqWsAPq/IvUFTvZLSqSBPVzsd9CmP0?=
 =?us-ascii?Q?46H2dm17YLnHydjgs1lTIB7ust6jZ3Kr6wXYTZ5gLe4P3xl2LA2/rUFwxu/t?=
 =?us-ascii?Q?ouaSs2jVeKfIvgwahVJhhwo2+ztIrMzNOp5xeLIHxPhr9wuSCBFQpObHKwpZ?=
 =?us-ascii?Q?ZsoalkW9G4/Tynffw+AbeG6CUWPcgmljW+UPoS6M2447A0qknMt5FfZ5BJr5?=
 =?us-ascii?Q?dshTcoIfvsnPX2d7H6IVlDKQ258rMBR9EKC3hKLOpK30slzCn9OweHkqIcPh?=
 =?us-ascii?Q?8YSGtcJX6U2qDT00HDCrzCVt52srQmVNKyeJa9qIvoBFU4+nyHLBncqtYxt+?=
 =?us-ascii?Q?RX1QjEUXaX4mTNcXvxFy8M5ZhU+AQsO1IDZxCyEHlc9MfCyK3r1oj+EqRYRp?=
 =?us-ascii?Q?Ox4+2bZ7l/s7NoDe+bUNqdId0NhoOIBjCqUzFg9yb8TUfIZbXRo+hGeXzPa9?=
 =?us-ascii?Q?etr6gw60zcJ92atMmMoM/ZMTIXSATvFxT79fsQZeZttMl6KVl6Vso49uKoD9?=
 =?us-ascii?Q?EzqEDEXZ2cdYDPdR7fVmj6Ibp+mikvJSOu0plrXeXAUQQb/bzA4CUIn0+Npj?=
 =?us-ascii?Q?MKC/o70zuT/wWiSVao/pWpcoWy+wMPlO3U7TdOiUT4Qu3qgA1ImqsdjNbAi/?=
 =?us-ascii?Q?XkAQn+kJH6gxY3vW3kU6J2PGuoJ6AOSyQKtFupITQQdCchIyCg7bxrAoPHfS?=
 =?us-ascii?Q?IBzM6QyUKI1tGk4VaU2YRws4n4msMoSVuhMRS/hymQ3NFnK2GHbygBkObJQy?=
 =?us-ascii?Q?U6UCMdhE/kEn1RQcTApalnO2W2k+OH4HigQ5SimkraEx/e7lgKWRghQmpgIq?=
 =?us-ascii?Q?6DRA3FkcSrz13yeVFJF2yymPrRAvRLbZXkTSfExfBfH5BgkeIuB/hwv3jCPg?=
 =?us-ascii?Q?OdZBZBXchpeK5P8ut9eTb1yhOKrVKNfNgTbltg+YMX1WzaBKRBfWhWFItqPX?=
 =?us-ascii?Q?0wOQeiSbzhNrIQ9eZNj7Zcgd2fgUpUxecioq/2eTv4jIL9wFxaSJBEvaZxJ/?=
 =?us-ascii?Q?x2BvPacMyS8mXvcWaG957CYfQlbvGF4CzPf0IG1KVY0tbohisFiKBAOah8Fs?=
 =?us-ascii?Q?+lltBmJGEmMaO5xmjbFeh8jDc/XUP4gFQkP62a+p7VSOMN4uP/HXwO+isFF5?=
 =?us-ascii?Q?1X8R5uEbGTf9db4C3A8pa1Dq4gtF5daMQbmp/FiS+ZaBO692TtFWESb/eu4d?=
 =?us-ascii?Q?Uh4pG6ZO3gLbFiJbvaX0nGlkM/D+9u8fMcre2E6zbgR+xi9VvKvZJaNQTlKf?=
 =?us-ascii?Q?xk2q4DxUDSnCX2Xl++Ijv6Fj6p8oxGiAmq8VhldtZSwGADl/XhdsBKXvkjyK?=
 =?us-ascii?Q?2voEiEIexkKWwoKh0wP5cfpNFd3UfFLO10n97dsirfvQX3U2e1Fg6U90NXrQ?=
 =?us-ascii?Q?MsnohVMGfENSTPCiVOs6ae70zNzYf7xvhAPIT1vzfuBpSng1coK0j3ILgumP?=
 =?us-ascii?Q?UTa9obqyDWTdctnQ81KkJzdxx5HLjK2Hpx0U7OOKXsgFjTNA1iZzJWNWLDKG?=
 =?us-ascii?Q?G1/LpTJDpnTBjDUZ6F0fXFrGZRw2SssRy4Trf8/o?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 646a035f-fe9e-475d-de65-08db25e6b55d
X-MS-Exchange-CrossTenant-AuthSource: CY5PR11MB6392.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Mar 2023 06:21:42.2183
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: hXTLxp5S4MwmI/sx4y27Tr0EhCng7w5Smq2hiqBFsKjho8jcA2kzafUE28QvgbhPgfZqNSvJ43lzLg1JG92Gcg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR11MB7400
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, Mar 06, 2023 at 03:20:09PM +0200, Adrian Hunter wrote:
> On 6/03/23 08:30, kernel test robot wrote:
> > Greeting,
> > 
> > FYI, we sent a report when this commit was in linux-next/master:
> > 
> > https://lore.kernel.org/all/202302092156.c0b14693-oliver.sang@intel.com/
> > 
> > This commit has been merged to mainline and we still observe similar
> > failures, so we report again FYI.
> > 
> > We noticed perf-sanity-tests.Test_data_symbol.fail due to commit (built with gcc-11):
> > 
> > commit: ce4c8e7966f317ef2af896778b5fa9105a5cd351 ("perf symbols: Get symbols for .plt.got for x86-64")
> > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> > 
> > [test failed on linux-next/master 1acf39ef8f1425cd105f630dc2c7c1d8fff27ed1]
> > 
> > in testcase: perf-sanity-tests
> > version: 
> > with following parameters:
> > 
> > 	perf_compiler: clang
> > 
> > on test machine: 8 threads 1 sockets Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz (Kaby Lake) with 32G memory
> > 
> > caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
> > 
> > 
> > 2023-02-08 19:29:00 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-func-ce4c8e7966f317ef2af896778b5fa91
> > 05a5cd351/tools/perf/perf test 105
> > 105: Test data symbol                                                : FAILED!
> > 
> > 
> > =========================================================================================
> > tbox_group/testcase/rootfs/kconfig/compiler/perf_compiler:
> >   lkp-kbl-d01/perf-sanity-tests/debian-11.1-x86_64-20220510.cgz/x86_64-rhel-8.3-func/gcc-11/clang
> > 
> > commit: 
> >   51a188ad8c2d8 ("perf symbols: Start adding support for .plt.got for x86")
> >   ce4c8e7966f31 ("perf symbols: Get symbols for .plt.got for x86-64")
> > 
> > 51a188ad8c2d89c5 ce4c8e7966f317ef2af896778b5 
> > ---------------- --------------------------- 
> >        fail:runs  %reproduction    fail:runs
> >            |             |             |    
> >            :6           50%           3:6     perf-sanity-tests.Check_branch_stack_sampling.fail
> >            :6          100%           6:6     perf-sanity-tests.Test_data_symbol.fail
> >            :6           33%           2:6     perf-sanity-tests.perf_record_tests.fail
> > 
> > 
> > If you fix the issue, kindly add following tag
> > | Reported-by: kernel test robot <yujie.liu@intel.com>
> > | Link: https://lore.kernel.org/oe-lkp/202303061424.6ad43294-yujie.liu@intel.com
> > 
> > 
> > To reproduce:
> > 
> >         git clone https://github.com/intel/lkp-tests.git
> >         cd lkp-tests
> >         sudo bin/lkp install job.yaml           # job file is attached in this email
> >         bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
> >         sudo bin/lkp run generated-yaml-file
> > 
> >         # if come across any failure that blocks the test,
> >         # please remove ~/.lkp and /lkp dir to run from a clean state.
> 
> Works fine for me.  See below.
> 
> You will have to debug your environment:
> Add option -v to perf test (e.g. perf test -c "data symbol") and
> add 'set -x' to tools/perf/tests/shell/test_data_symbol.sh

Thanks for the suggestion on debugging our environment.

We found that we were asked by developers to run perf test with
sanitizer on, so we build perf with some additional flags like
`make EXTRA_CFLAGS="-fsanitize=undefined -fsanitize=address"`, and the
test_data_symbol failure only happens when these flags are set, but
won't happen if do a make without those flags.

Could you please help check if above flags could impact the behavior of
perf test? Thanks.

Here are the test results with and without the flags:

$ make EXTRA_CFLAGS="-fsanitize=undefined -fsanitize=address"
$ ./perf test "data symbol" -v
105: Test data symbol                                                :
--- start ---
test child forked, pid 25107
+ skip_if_no_mem_event
+ perf mem record -e list
+ grep -E -q available
+ return 0
+ TEST_PROGRAM='perf test -w datasym'
++ mktemp /tmp/__perf_test.perf.data.XXXXX
+ PERF_DATA=/tmp/__perf_test.perf.data.t834P
+ trap cleanup_files exit term int
+ echo 'Recording workload...'
Recording workload...
++ grep -E -c 'vendor_id.*AuthenticAMD' /proc/cpuinfo
+ is_amd=0
+ (( 0 >= 1 ))
+ PERFPID=25120
+ sleep 1
+ perf mem record --all-user -o /tmp/__perf_test.perf.data.t834P -- perf test -w datasym
util/util.c:527:2: runtime error: null pointer passed as argument 2, which is declared to never be null
+ kill 25120
+ wait 25120
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.154 MB /tmp/__perf_test.perf.data.t834P (993 samples) ]
+ check_result
++ perf mem report -i /tmp/__perf_test.perf.data.t834P -s symbol_daddr -q
++ awk '/buf1/ { print $4 }'
+ result=
+ '[' -z '' ']'
+ return 1
+ exit 1
+ cleanup_files
+ echo 'Cleaning up files...'
Cleaning up files...
+ rm -f /tmp/__perf_test.perf.data.t834P
test child finished with -1
---- end ----
Test data symbol: FAILED!


$ make
$ ./perf test "data symbol" -v
105: Test data symbol                                                :
--- start ---
test child forked, pid 17241
+ skip_if_no_mem_event
+ perf mem record -e list
+ grep -E -q available
+ return 0
+ TEST_PROGRAM='perf test -w datasym'
++ mktemp /tmp/__perf_test.perf.data.XXXXX
+ PERF_DATA=/tmp/__perf_test.perf.data.qqdDQ
+ trap cleanup_files exit term int
+ echo 'Recording workload...'
Recording workload...
++ grep -E -c 'vendor_id.*AuthenticAMD' /proc/cpuinfo
+ is_amd=0
+ (( 0 >= 1 ))
+ PERFPID=17247
+ sleep 1
+ perf mem record --all-user -o /tmp/__perf_test.perf.data.qqdDQ -- perf test -w datasym
+ kill 17247
+ wait 17247
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.364 MB /tmp/__perf_test.perf.data.qqdDQ (5039 samples) ]
+ check_result
++ perf mem report -i /tmp/__perf_test.perf.data.qqdDQ -s symbol_daddr -q
++ awk '/buf1/ { print $4 }'
+ result='buf1+0x0
buf1+0x38
buf1+0x38
buf1+0x0'
+ '[' -z 'buf1+0x0
buf1+0x38
buf1+0x38
buf1+0x0' ']'
+ IFS=
+ read -r line
+ '[' buf1+0x0 '!=' buf1+0x0 ']'
+ IFS=
+ read -r line
+ '[' buf1+0x38 '!=' buf1+0x0 ']'
+ '[' buf1+0x38 '!=' buf1+0x38 ']'
+ IFS=
+ read -r line
+ '[' buf1+0x38 '!=' buf1+0x0 ']'
+ '[' buf1+0x38 '!=' buf1+0x38 ']'
+ IFS=
+ read -r line
+ '[' buf1+0x0 '!=' buf1+0x0 ']'
+ IFS=
+ read -r line
+ return 0
+ exit 0
+ cleanup_files
+ echo 'Cleaning up files...'
Cleaning up files...
+ rm -f /tmp/__perf_test.perf.data.qqdDQ
test child finished with 0
---- end ----
Test data symbol: Ok


--
Best Regards,
Yujie