Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <823b8819-b808-d11c-acbc-9caff9e2f535@amd.com>
Date:   Mon, 7 Nov 2022 15:59:47 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.3.3
Subject: Re: Perf: BUG: kernel NULL pointer dereference, address:
 0000000000000198
To:     Peter Zijlstra <peterz@infradead.org>
Cc:     Naresh Kamboju <naresh.kamboju@linaro.org>,
        Arnaldo Carvalho de Melo <acme@kernel.org>,
        Borislav Petkov <bp@alien8.de>, Ingo Molnar <mingo@redhat.com>,
        linux-perf-users@vger.kernel.org,
        open list <linux-kernel@vger.kernel.org>,
        regressions@lists.linux.dev, lkft-triage@lists.linaro.org,
        Stephane Eranian <eranian@google.com>,
        Sandipan Das <sandipan.das@amd.com>,
        Ravi Bangoria <ravi.bangoria@amd.com>
References: <CA+G9fYtOP-spGS7gYyBB-8wpUDiKxd5e_Nk05J9gzWR_DLBeFA@mail.gmail.com>
 <4429582f-155a-04d2-d139-a834bb325612@amd.com>
 <d577887d-b86b-82be-8081-f7bebacd8067@amd.com>
 <5bec4620-2e2b-abe0-1e82-528c084d4d16@amd.com>
 <cb40c209-9c4c-f645-8454-bcd74cb72405@amd.com>
 <Y2O1GDRrJqIu17vM@hirez.programming.kicks-ass.net>
 <f00ecc51-a387-1801-0db6-e18060c620a7@amd.com>
Content-Language: en-US
From:   Ravi Bangoria <ravi.bangoria@amd.com>
In-Reply-To: <f00ecc51-a387-1801-0db6-e18060c620a7@amd.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cXVmenY0YnNvYlpPK2crQ3ozRnMva2M0R21sKzNtZVNNVEc3VlpSbWcxZVl5?=
 =?utf-8?B?dkI5SE41NlhFVmdEVHlDdHRwOXFQdCtKWlBpZ1JkVDA0ayt6Q3JMdWVjU0Ja?=
 =?utf-8?B?bmhIYVhXMTR1UG9yYVphMzcxWmhYMnp0d1UrU1EvNDVCM1crY255S2xwM3pL?=
 =?utf-8?B?Tzl4YjVNMmpiZURqa0JWRHVzc1dLbHo2SEpnOE9DeDFYa1M0RFJYSDZ6QTVu?=
 =?utf-8?B?T0ZwOW5neTJSZEh6R3JrYS9EOFZxRVUyN0s0WUV6MUZVQ1VsR3UxcEphWm55?=
 =?utf-8?B?WGhJbDlHSDVYNy9xN1d5YkcvTURKRVFWZUJHVTBtSUc1VjJkd0o3REI1RzBy?=
 =?utf-8?B?V2haSmc3RVlpU2lPVWoyVEw1dnZ1WFlVTy83R0ljOWRSTmU4K2M4YXhoYVFP?=
 =?utf-8?B?d0l0OTRBcFVBdHY1K2NBYk1Cd2xXSVBOdDZKdU5sSjlGODQzbUxOc2xaNEdQ?=
 =?utf-8?B?bnIrWTdIKzZpK2ozaGVMMUNpWmkxSEs2RTU0OW53ZllqS2N6VjdaeTM2eG1E?=
 =?utf-8?B?Z2pxd0N6M283cXVGdEhnVU56bElMMFQwRDUxazdDUnZCM1pJSUdUMzg1dy83?=
 =?utf-8?B?T0ZOalVpVWUxTEM3Z1dzOXFLM2dWTUdJbkJPTlQrclQxTS9McEpET0t2UU5D?=
 =?utf-8?B?ZFZ1bzVnbFJkWnhwR2RJSHhWY1VvVHRHZnB0NzNiQ2NGNm96QmNDeGw5c1ZS?=
 =?utf-8?B?SG1YeDFYMEJvWFVBaWp3d3REWWtIZ1FoL1BIVGVpZmhrVTN2NHR0OUxVTGRa?=
 =?utf-8?B?WlI3dWhoS3E3T2ROckFvOEN3STVHdXhZTDlyZTh2d210VDJpUzB4dVhxNUVM?=
 =?utf-8?B?amk4b05Vbkg5SVFIREVkNmNtcUNEZHgwbFNjSy81ZUprbDVrYnZmeDhpRGpx?=
 =?utf-8?B?QW05RFFUNEVkZGpYNVdzazJxbmtROWxhSG16cXllT3g2LzdCS1JzM3hzYnpr?=
 =?utf-8?B?bVpmNThlNnRaZjFhZHAvcHBiSWlseXFiQmxFMUdUU2lMTU5XdXpFNW1hZUdw?=
 =?utf-8?B?WDR2MUsxL3FDWS91QkxOa2FpM2syZEFkZDEvdlozTGlXb1gvYW9iUmNVSlN0?=
 =?utf-8?B?UlM3K3p3QkdUTlJLNXRxTFpucW9NakQ5aFB1bWVac2h0cTdBOVlhQUJUbkNP?=
 =?utf-8?B?U2FGaUxiWTJTUWl0UDFsT08rYnozMG1rbkc5V251azk5dG1YWjlqNVdnVmV5?=
 =?utf-8?B?Z1hBNmNWLzdrTSt1bG9rd1Brc0VoTUNvNndTdEZmT0NYMUFVaWE2cW5zbHBk?=
 =?utf-8?B?YzVIWUIzVk9YNis5K2NTOW1IMVZiQnE0SUhXdkdrTnZ0Y0JVT3dVL0lQa0di?=
 =?utf-8?B?TEpuR1FJLysyNzhsbWM0MUtIeUplMkhWa0dib0pOU091TVUxdTV0ODA0QUJE?=
 =?utf-8?B?Wk9hbjJkUnViQU9GUlovYU56UWJaWFlJQys1dHFyUU9HRkc3SWgwOXVhVURT?=
 =?utf-8?B?Y3N4Sk8yVFJSb2EwM2svOHdSL01BWU4vMnhIZWxJcUF3WEc5RFFrdlJxZ3dl?=
 =?utf-8?B?Z1RkaGRxbmhiSm9mVG0ySEVadFVzUFdEVEFpc2dnK3FxYytRQmRyRXZZNGxR?=
 =?utf-8?B?MHFjZGpoVjBxNWlJZWM2d0o5a0t6SklwVXFIS0d5d1phSzNuekljcFpmV0dz?=
 =?utf-8?B?OVNmalBaRDdIWUpoVGVVRExwYzVQNlVtbkJxRGJtV0FZcFpYckIrWnZ2VlVv?=
 =?utf-8?B?MXlnN1hJN0h3US9qNVpObCtOSlgwMUFwMW9nZGZKQU5IR0ZySitnSDNnL1pk?=
 =?utf-8?B?NmRLdWJBWnZJMzV2WjV0QWkrc2RwNSsrNG45UDVsYVhmK0gxL2RFZnh5MUhH?=
 =?utf-8?B?Rk51TUpraDN0SE5jWS9sSW9UdWhwTE9XT3FxdForQWgyejFuY09xWUc2cnZH?=
 =?utf-8?B?TTdWZjlwSDRzb1hEWXhSRWw4eHJ3QkIzR01uTmtQSkI4VHhYcDRFMzh4ZWls?=
 =?utf-8?B?WjRzak1KWlhUYUlHaFRUMjJsRGlPcWNIR0t4M3hPWEFNelZkZkVybTd2TXYz?=
 =?utf-8?B?SmRPeVkvckw0aUpKMHlCQ3VCMXRqZU5NQ3Q5eXF2Ymk2WE1ITDJaTzNJM1Ix?=
 =?utf-8?B?SC8wWmZzNnZ0OTBTTTJKbkVPYkNZRFZydVJleEhtL2ZGNGxQVlNtZXpJajNl?=
 =?utf-8?Q?XSUhe6L8whE+SD/Sm1xMVD45D?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 3a02a0f1-cefb-4677-34e4-08dac0ab068b
X-MS-Exchange-CrossTenant-AuthSource: PH7PR12MB6588.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Nov 2022 10:30:01.1056
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: jV2Ij76jgNMXzLk4CiODEiYVMdo0Ztfu5NmYtfyV1m7yqC927eg3V4w3tNSTVayfEoJqk9nZvKcH9uRSuinXMw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4520
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 03-Nov-22 9:42 PM, Ravi Bangoria wrote:
> On 03-Nov-22 6:03 PM, Peter Zijlstra wrote:
>> On Thu, Nov 03, 2022 at 05:15:30PM +0530, Ravi Bangoria wrote:
>>
>>> Sorry was distracted a bit. So, this seems to be happening because of
>>> race between amd_pmu_enable_all() and perf event NMI. Something like:
>>>
>>>   amd_pmu_enable_all()
>>>   {
>>>       if (!test_bit(idx, cpuc->active_mask))
>>>
>>>   --->/* perf NMI entry */
>>>       ...
>>>       x86_pmu_stop() {
>>>           __clear_bit(hwc->idx, cpuc->active_mask);
>>>           cpuc->events[hwc->idx] = NULL;
>>>       }
>>>       ...
>>>   <---/* perf NMI exit */
>>>
>>>       amd_pmu_enable_event(cpuc->events[idx]);
>>>   }
>>
>> Hmm, do you have more information?
> 
> I've extracted function graph logs from crash dump and uploaded it here:
> https://github.com/BangoriaRavi/function_graph/blob/main/trace.function_graph.dat
> 
> crash was on CPU1.

git bisect let me to a BRS patch:

    commit ada543459cab7f653dcacdaba4011a8bb19c627c
    Author: Stephane Eranian <eranian@google.com>
    Date:   Tue Mar 22 15:15:07 2022 -0700

    perf/x86/amd: Add AMD Fam19h Branch Sampling support

    Add support for the AMD Fam19h 16-deep branch sampling feature as
    described in the AMD PPR Fam19h Model 01h Revision B1.  This is a model
    specific extension. It is not an architected AMD feature.

Thanks,
Ravi