Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <bf8a41ed-3d93-9f60-5e2d-780abc177325@oracle.com>
Date:   Tue, 22 Mar 2022 16:00:43 -0400
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.7.0
Subject: Re: [PATCH] xen: fix is_xen_pmu()
Content-Language: en-US
To:     Juergen Gross <jgross@suse.com>, xen-devel@lists.xenproject.org,
        x86@kernel.org, linux-kernel@vger.kernel.org
Cc:     Stefano Stabellini <sstabellini@kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        "H. Peter Anvin" <hpa@zytor.com>,
        =?UTF-8?Q?Marek_Marczykowski-G=c3=b3recki?= 
        <marmarek@invisiblethingslab.com>
References: <20220322155001.21979-1-jgross@suse.com>
From:   Boris Ostrovsky <boris.ostrovsky@oracle.com>
In-Reply-To: <20220322155001.21979-1-jgross@suse.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UlovNXRYUlYvMWtoVWg1dktqam5tZHY4ZkprbWNtQnJTUEVlSTY0RndyRFVn?=
 =?utf-8?B?SUtVckZLYXFHdDhQeGkrMFdJWE5UZS9IeXUweTdrSkNQUU93a21qN0tZU3hS?=
 =?utf-8?B?M2xvOWtsQ2tVc2I3WXVjekJVa0JwbkZWdVpOaFp2SWJXMVo5RktaczhVaVg4?=
 =?utf-8?B?T3NNVTNjZHJYR2w2MTFGbXdWMU5DZUl6cFBTRzA2VGhicmtnTXZZeGtqZ3F6?=
 =?utf-8?B?d2FNaFdLa3poVjlCVEFDZUFRWjduOTlRZlFzUnVPWUtRVHNlQ05UVnNnOUc1?=
 =?utf-8?B?ZUlnRnVxVys2blUxZnlMeEYrZ1lTdm1MWlVyTlprYlpzcTNKVVBnd3RoWHJN?=
 =?utf-8?B?d1FVTDM1c3ZJZVlEK0h1d3VaK2JwUE1WcENpbG9JMVRyQS9yUXpSSk0yMXVX?=
 =?utf-8?B?bkdIN1FLNE5uR0VMdDlqWVVuWGxWbzZhQnR0SEREQ1l5M0dWYklOUmZjSmhx?=
 =?utf-8?B?bUs2cnZ0cXB0SEJqb2dFbWZuUUhsUTZORDZMTGxKdllFTWZ3dW9WTTUvS0J1?=
 =?utf-8?B?YkcyV3ZSSVE2L1RBYkF0RTlLRVhOVURvNisxM2ljOERkK1VmZ2c0cVo3UlFi?=
 =?utf-8?B?Qy9temlFMndxaCtxLzVkUWNtWDNkeVNEdGY5SlFmNk5wK2RqaTBpdDIxbjl6?=
 =?utf-8?B?eVlTVTBlVVV4cExyVW96Ty8wVXZjTktMbUpmc3BiaWM3c0tINjBEcndkVVg3?=
 =?utf-8?B?dzMzUEpJV3VFOXhTbk11SlNZR3Fsbk5CWE5vQkZTbXQ5aHNMT2crSFFLeU5J?=
 =?utf-8?B?OUlQZjltRmVQVkk1ZnRhaEVvZ0JHQXBuSDFFZUxNaVlQV2dGN3RmSHJ4K0JB?=
 =?utf-8?B?SzRyOG9ZTm5ibXFtV1NtbEZtVk5YcUNHRXNkaDlLM0orMDU2YlorUDBJemxH?=
 =?utf-8?B?YlFCUUhpcWpndkdOcHlaRENSVThRaEZBKy8xZEEzaDY5RHlWTHJWSU8zVUJs?=
 =?utf-8?B?Z3JhTjNJVDkrRUNuZ0R2c0NFbkk2em1YV243eXcwbHJZemxaUXpOelJZNW01?=
 =?utf-8?B?ZGtlZzFPTXN1Yk01a0VqNnZ0aHUxQ1dBYk02YnRnTFVUMFRGSDM5UkV6Z01n?=
 =?utf-8?B?cFJkbWh1eFBPaXpscWx2dDdSdVNzOFNqWEYxZzN0UEhRQjZlS1g5eW1pNnJz?=
 =?utf-8?B?RXNzUitBQ3ZldDRBdVQvS2REeHpBRVZqSVpvUkZSUy9vdGQ2OGJzM1o0aGpj?=
 =?utf-8?B?Rksza0REb3FUZlpscW8wVW9abEZZdWtJMG5ld1VvQ242aVN5U09lY2szTWpC?=
 =?utf-8?B?WksvRW8vY0UvdTJDazdtWk9EKzhTRU5SdDFJZGFnWXNjWFNtZll0dVFuM1I3?=
 =?utf-8?B?SDlENFNsZ2JHY3dWUzYxb0hyVGt2UW1YRzJwNDM2MVUycmZzL2ZvWXhCOTUr?=
 =?utf-8?B?dGN3ZTBiRVdET012UjJ3bWkzTkFOMmEvZUovTzVGUzZVRmVWRXNJM2ZKMmt3?=
 =?utf-8?B?ZVJYaXB1SXZ5OG5wSFc0dGV4L2k2K2lzc0o1dENiODFjZlJLVU5pUElWQllv?=
 =?utf-8?B?bGFlMjJ1djMzQTRURU1XZzBrZXNnVTQ3YXptYitQMTBiTEYxaXlrcGNtSGQ2?=
 =?utf-8?B?RlRZdFBEMERjQnRqdEJ4UXZnc1gyQkRwQ1JmTWhQUzRoK01vVVdVWSt6a0d0?=
 =?utf-8?B?M3A5WlIvalVSRk8wb3kySHM1L0NYamNMdisrajcxeURnOWlkbjJzNHJERkNN?=
 =?utf-8?B?dk5tMU5abnI3RWVJS09iWTl5cDg2dGY4ODU0dWdMUnUzTGNzUExsRU1pYlF6?=
 =?utf-8?B?VFV3RnlxV1JmRGhVUXUwVG9HUDFFUURVdWpRU29BV1p5R0JVZURWT0x1UWJ2?=
 =?utf-8?B?MWtHcm5mMEdNdUFxdVk0SHpxU0JVUDNlbkxrMmc5RUZMZmxjZDhTcmszVG1M?=
 =?utf-8?B?eXZuUjNGeEdSRHM5NXloVFF3UUx3SldLWmZmMSs5Skk5cTRsL0I4VksxcUxV?=
 =?utf-8?B?d0dpS0VNMmZOdnJqUXBYWUlNcWFXQmxPQ08wQU04MmFuQmFTQU1MWHpsRGFU?=
 =?utf-8?B?Y1NIUndLWDdRPT0=?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 24e44fd6-698c-4d94-be61-08da0c3eaa2b
X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB5009.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2022 20:00:50.9939
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 59BcXluEotDzo3o4rssxOAdbHSJelYaIbypuy3wMRlELB6s6bUziJ2Q5akZVP+iW8Voe3+Q8Y+RFFF3eRpBBT08jZtWZ9KMCSyXaTJMQFYs=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR10MB2730
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10294 signatures=694350
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 phishscore=0
 mlxlogscore=999 adultscore=0 suspectscore=0 malwarescore=0 spamscore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2202240000 definitions=main-2203220104
X-Proofpoint-GUID: iaul_O4QrRAbRmSyqt_mwBeOGpf-C1BU
X-Proofpoint-ORIG-GUID: iaul_O4QrRAbRmSyqt_mwBeOGpf-C1BU
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 3/22/22 11:50 AM, Juergen Gross wrote:
> is_xen_pmu() is taking the cpu number as parameter, but it is not using
> it. Instead it just tests whether the Xen PMU initialization on the
> current cpu did succeed. As this test is done by checking a percpu
> pointer, preemption needs to be disabled in order to avoid switching
> the cpu while doing the test. While resuming from suspend() this seems
> not to be the case:
>
> [   88.082751] ACPI: PM: Low-level resume complete
> [   88.087933] ACPI: EC: EC started
> [   88.091464] ACPI: PM: Restoring platform NVS memory
> [   88.097166] xen_acpi_processor: Uploading Xen processor PM info
> [   88.103850] Enabling non-boot CPUs ...
> [   88.108128] installing Xen timer for CPU 1
> [   88.112763] BUG: using smp_processor_id() in preemptible [00000000] code: systemd-sleep/7138
> [   88.122256] caller is is_xen_pmu+0x12/0x30
> [   88.126937] CPU: 0 PID: 7138 Comm: systemd-sleep Tainted: G        W         5.16.13-2.fc32.qubes.x86_64 #1
> [   88.137939] Hardware name: Star Labs StarBook/StarBook, BIOS 7.97 03/21/2022
> [   88.145930] Call Trace:
> [   88.148757]  <TASK>
> [   88.151193]  dump_stack_lvl+0x48/0x5e
> [   88.155381]  check_preemption_disabled+0xde/0xe0
> [   88.160641]  is_xen_pmu+0x12/0x30
> [   88.164441]  xen_smp_intr_init_pv+0x75/0x100


There is actually another PMU-related problem on restore which was caused (or, rather, highlighted) by ff083a2d972f56bebfd82409ca62e5dfce950961:


[  116.861637] ------------[ cut here ]------------
[  116.861651] WARNING: CPU: 1 PID: 31 at kernel/events/core.c:6614 perf_register_guest_info_callbacks+0x68/0x70
[  116.861673] Modules linked in:
[  116.861682] CPU: 1 PID: 31 Comm: xenwatch Not tainted 5.17.0-rc7ostr #103
[  116.861695] RIP: e030:perf_register_guest_info_callbacks+0x68/0x70
[  116.861706] Code: c7 c7 40 e1 86 82 e8 d7 e7 ff ff 48 8b 53 10 48 85 d2 74 14 48 c7 c6 f0 0a c0 81 48 c7 c7 30 e1 86 82 5b e9 ba e7 ff ff 5b c3 <0f> 0b c3 0f 1f 44 00 00 0f 1f 44 00 00 48 8b 05 54 fd 0b 02 48 39
[  116.861747] RSP: e02b:ffffc9004016fe18 EFLAGS: 00010286
[  116.861758] RAX: ffffffff82432850 RBX: 0000000000000003 RCX: ffff888079c00000
[  116.861768] RDX: ffff888079c00000 RSI: ffffc9004016fe30 RDI: ffffffff82432850
[  116.861778] RBP: 0000000000000000 R08: 0000160000000000 R09: ffffea00000ed340
[  116.861788] R10: 0000000000000758 R11: 0000000000000000 R12: ffff888003b4d000
[  116.861797] R13: 0000000000000003 R14: ffffffff8162cf10 R15: 0000000000000000
[  116.861819] FS:  0000000000000000(0000) GS:ffff888079c80000(0000) knlGS:0000000000000000
[  116.861830] CS:  e030 DS: 0000 ES: 0000 CR0: 0000000080050033
[  116.861839] CR2: 0000000000000000 CR3: 00000000062b6000 CR4: 0000000000040660
[  116.861853] Call Trace:
[  116.861861]  <TASK>
[  116.861866]  xen_pmu_init+0x187/0x280
[  116.861879]  xen_arch_resume+0x30/0x50
[  116.861888]  do_suspend.cold+0x132/0x147
[  116.861899]  shutdown_handler+0x12e/0x140
[  116.861910]  xenwatch_thread+0x94/0x180
[  116.861919]  ? finish_wait+0x80/0x80
[  116.861928]  kthread+0xe7/0x110
[  116.861938]  ? kthread_complete_and_exit+0x20/0x20
[  116.861948]  ret_from_fork+0x22/0x30
[  116.861959]  </TASK>
[  116.861964] ---[ end trace 0000000000000000 ]---


I was going to send a patch but I think yours can be slightly modified to take care of this problem as well.


> @@ -542,6 +539,7 @@ void xen_pmu_init(int cpu)
>   	per_cpu(xenpmu_shared, cpu).flags = 0;
>   
>   	if (cpu == 0) {


               if (!is_xen_pmu)


> +		is_xen_pmu = true;
>   		perf_register_guest_info_callbacks(&xen_guest_cbs);
>   		xen_pmu_arch_init();
>   	}
> @@ -572,4 +570,7 @@ void xen_pmu_finish(int cpu)
>   
>   	free_pages((unsigned long)per_cpu(xenpmu_shared, cpu).xenpmu_data, 0);
>   	per_cpu(xenpmu_shared, cpu).xenpmu_data = NULL;
> +
> +	if (cpu == 0)
> +		is_xen_pmu = false;


And drop this hunk.


-boris