Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <8e3e4716-22e7-8684-394c-b5a457906975@oracle.com>
Date:   Mon, 28 Feb 2022 15:45:13 -0500
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.6.1
Subject: Re: [PATCH v3 0/1] xen: fix HVM kexec kernel panic
Content-Language: en-US
To:     Dongli Zhang <dongli.zhang@oracle.com>,
        xen-devel@lists.xenproject.org, x86@kernel.org
Cc:     linux-kernel@vger.kernel.org, jgross@suse.com,
        sstabellini@kernel.org, tglx@linutronix.de, mingo@redhat.com,
        bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com,
        joe.jin@oracle.com
References: <20220224215049.2845-1-dongli.zhang@oracle.com>
 <9de1059b-6b48-e2c8-60bc-f29b42b217f7@oracle.com>
 <36014dbd-e63c-cf49-dafd-2d7a8b84db5d@oracle.com>
From:   Boris Ostrovsky <boris.ostrovsky@oracle.com>
In-Reply-To: <36014dbd-e63c-cf49-dafd-2d7a8b84db5d@oracle.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?K0ZtUVVSZDFLeUtXVFRjWHBBdVppbjVZcWpTVitGaHdva1pVN2JObEtnbkg4?=
 =?utf-8?B?NW9udGovcUIxcVNQOUtmTy9PMnpIbCtveTJtakVUczhlcmMyLzBQdGFoeEdn?=
 =?utf-8?B?NjR3OXE1bVZrUnMzcG1oK29BT0xzQzlyZXRPQzhOa2xTRzdueXI3QUt1SHEv?=
 =?utf-8?B?eEtXV3VBelhEQlR2cXJ2TTE0bkFCeDBzb2NvTFRNdDgwR2dCWFNJL1VLdkxU?=
 =?utf-8?B?MWRHRUMyZ0tkZEJpMTNJdHg5Z0QwMlpIUDBRQkxrSElzV2g3S2x0czFFalUv?=
 =?utf-8?B?aVM0Q2JuWlo0WWN5bmRiRDZBYjMrb1RIOGJqWlFvcXgyRDhrVXVRMWd2QXlQ?=
 =?utf-8?B?emx2NTlRWDRFZGd3YmxKWFVBMWpuMG5Oek1zZzZjL2pzQ001bzY3aXdQNnlO?=
 =?utf-8?B?bjdvZUdLemQ3TlAzdHV6bFhGNEErRWxOVmxOd0xETWF1Tk9FV2Q2Ri9HRmE4?=
 =?utf-8?B?WUJLVlh3OG1Zc0ZYWkcvSW92M21GWXlnY08relcxTG82RkFLeFF1RXRNSi81?=
 =?utf-8?B?TFhGYlY5TVhaajJZaW05dWdyZnYvN09LUWFBaWt2QUsxQ0hzbTVpdEU3MmZZ?=
 =?utf-8?B?ZGNCamV0UE9BajQvN0NXRUZkMUJ3QVBlK1ZEVmkrMDlJbHhwZHk0b1hONlND?=
 =?utf-8?B?OTJaemoxNnpHZjZXSVBpY0pLM0JpekZyTmlDRjJjUmZWRnZFaDJNK0R0Q2xw?=
 =?utf-8?B?WkJIM3pQRmYxTmhIWUF2Y3VtYzNVc2RtZ2ZSYmtNc1JHcE5NV2hZMVRWUm1r?=
 =?utf-8?B?Z3l4QXlRMWJJOWZmeUZPWkJqbnYvbWZxSC92TE9ZYW9sRU5PSG9hc0k3M0hx?=
 =?utf-8?B?VTJIRW5MRjdZb3Y4SVJpdlcxVnB0SWdhTEJBa2lGZ1dqcWljbFFHMzhUdTJD?=
 =?utf-8?B?UEtQelJINVc3UHFOdDlielh6ZldTN2Z0TlBjdkNKOXhjWHZVcEtyak9qOHlP?=
 =?utf-8?B?VlArenBqUUJ3N2VDVUFYUzgyVXZWaXh6enlaREcyM2NGY0JsRjRUZzJWemQw?=
 =?utf-8?B?MWsxc0hhSjlnQ08ycDVvOTFWalFYczlJNkxLamFpcW5abUVydjFWNktsdkFR?=
 =?utf-8?B?QmFpTUxDcExlUnMzcVlUeHd1amtKeCtFZWRqbXBySWFJUFlHQjhNbXRoOFpF?=
 =?utf-8?B?Y25lU0FvL3puVmpkWkM0SWRacHVtcG4zU2o1cG5YbjVEdUxmMXczcVg0dVMr?=
 =?utf-8?B?MlpzajdXMkRhN1NIZ015S2RPNXpScENIcHpDU0ExSVVvYzd2bWZRNWhKLzA2?=
 =?utf-8?B?N2xnYVo3bG9jQXQxNTJYUURqOThINDIzV3k5aHVKSUhhQ2RTYW1IR2pOWHRh?=
 =?utf-8?B?Sm1OQ2g4VWt4R1hrdUNsNDdyM1pEOGt0d1ovYmdqbjIzRXU3YTlHTklWaHhM?=
 =?utf-8?B?S0xxbEtCcGVnTVhDL2xDdWtNU21BVG9BL3dlYTVKK2grOXY1MG9vMEZYOTdi?=
 =?utf-8?B?eFFhMHhYZkdtWXZEYVIwMVJQTm5pa1V1SldacW5RMFVoZ01NRmIzV2xFYWE2?=
 =?utf-8?B?QkZsaXNKdjl5dDRveWt6UlNQNms0c2tPOFFaS0oyZnd5N0tJZzhrd3k1TW5n?=
 =?utf-8?B?ZXZhRnJIWFA0TjFWSkhYV1U5TkdjN3RueDZWVWUxWk16NmpZUDR6MjZzVXlW?=
 =?utf-8?B?dW02eUQwWnJZbnN1Wkpwd3VYWGpWSEZtN3IrdTVISTFFYzJkUlRkZ0NGeVpk?=
 =?utf-8?B?elMzTHVaSVdhZFpLNnVxUHR5bElmeXAxRTJUK0U0U3RFL2Y1Vit6OGtXeFJJ?=
 =?utf-8?B?NVU3NHpGcEw3cHpSLzU0YitqODZtMHhFOWdLNFR0bTZVcHJLeVlKMEVrbmoy?=
 =?utf-8?B?dXA4NE93TDA1Q05oNTVQdlViM3l1RmxpMTByc0czbGhoRlU1YktHcVlETTJQ?=
 =?utf-8?B?QW9WNnc0alpvNFdHL1Z2YlBtNks2ZkJOcjNmV2tyV3RvOU81ajdVWFFmUUwz?=
 =?utf-8?B?bE95dEZ6ZEFFcW1URGZTeFBnM0FReUpSQU9lMk13RFc5d1pQNGttR0JIWkUv?=
 =?utf-8?B?U0w4Mkx2SjR2R1VPOTJpTFJobVd1OFNFQ3JiQWRNRjBqelBlWHhXV2I2Q09J?=
 =?utf-8?B?L1FuWjRzRzhqR0JFZ3N4RU52QTZud2l2SlhZYnAxcHQ3aGI1bVA3K3lsejhy?=
 =?utf-8?B?ZTJQMTZjWERyYjdsK243b2tpbU1MYWs0Y25GVXZlWGp5SFFvdFdUSnkyTkxH?=
 =?utf-8?B?ZkpRU21QNk5pVzNVVDE3emgrZUdKOVFSVlJUU2xlSFpNbm0vWEt1c2luQW4w?=
 =?utf-8?Q?TkGTEYnQIyKWn6E38fjz+Ewm8RJjbM77yNjsac6Kns=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5b4b2d61-127c-4e75-d85b-08d9fafb3a83
X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB5009.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Feb 2022 20:45:17.7589
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: U6IHEUtWpBtFsXlTcvByx0SHxwtzSVVbZ7lhxcSPEi16aAwjp+7zH2a6GL31ogOMnzw531dmx1dajwF+oAk01o1lKd8rLpMMeT54uvP1SU4=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR10MB1999
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10272 signatures=684655
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 bulkscore=0
 adultscore=0 phishscore=0 spamscore=0 suspectscore=0 malwarescore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2201110000 definitions=main-2202280103
X-Proofpoint-ORIG-GUID: aIpQspA_c5pa9-fSlnkase4cvay-cBy2
X-Proofpoint-GUID: aIpQspA_c5pa9-fSlnkase4cvay-cBy2
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 2/25/22 8:17 PM, Dongli Zhang wrote:
> Hi Boris,
> 
> On 2/25/22 2:39 PM, Boris Ostrovsky wrote:
>>
>> On 2/24/22 4:50 PM, Dongli Zhang wrote:
>>> This is the v3 of the patch to fix xen kexec kernel panic issue when the
>>> kexec is triggered on VCPU >= 32.
>>>
>>> PANIC: early exception 0x0e IP 10:ffffffffa96679b6 error 0 cr2 0x20
>>> [    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted
>>> 5.17.0-rc4xen-00054-gf71077a4d84b-dirty #1
>>> [    0.000000] Hardware name: Xen HVM domU, BIOS 4.4.4OVM 12/15/2020
>>> [    0.000000] RIP: 0010:pvclock_clocksource_read+0x6/0xb0
>>> ... ...
>>> [    0.000000] RSP: 0000:ffffffffaae03e10 EFLAGS: 00010082 ORIG_RAX:
>>> 0000000000000000
>>> [    0.000000] RAX: 0000000000000000 RBX: 0000000000010000 RCX: 0000000000000002
>>> [    0.000000] RDX: 0000000000000003 RSI: ffffffffaac37515 RDI: 0000000000000020
>>> [    0.000000] RBP: 0000000000011000 R08: 0000000000000000 R09: 0000000000000001
>>> [    0.000000] R10: ffffffffaae03df8 R11: ffffffffaae03c68 R12: 0000000040000004
>>> [    0.000000] R13: ffffffffaae03e50 R14: 0000000000000000 R15: 0000000000000000
>>> [    0.000000] FS:  0000000000000000(0000) GS:ffffffffab588000(0000)
>>> knlGS:0000000000000000
>>> [    0.000000] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [    0.000000] CR2: 0000000000000020 CR3: 00000000ea410000 CR4: 00000000000406a0
>>> [    0.000000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>> [    0.000000] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>>> [    0.000000] Call Trace:
>>> [    0.000000]  <TASK>
>>> [    0.000000]  ? xen_clocksource_read+0x24/0x40
>>
>>
>> This is done to set xen_sched_clock_offset which I think will not be used for a
>> while, until sched_clock is called (and the other two uses are for suspend/resume)
>>
>>
>> Can we simply defer 'xen_sched_clock_offset = xen_clocksource_read();' until
>> after all vcpu areas are properly set? Or are there other uses of
>> xen_clocksource_read() before ?
>>
> 
> I have tested that below patch will panic kdump kernel.
> 


Oh well, so much for that then. Yes, sched_clock() is at least called from printk path.


I guess we will have to go with v2 then, we don't want to start seeing time going back, even if only with older hypervisors. The only thing I might ask is that you roll the logic inside xen_hvm_init_time_ops(). Something like


xen_hvm_init_time_ops()
{
	/*
	 * Wait until per_cpu(xen_vcpu, 0) is initialized which may happen
	 * later (e.g. when kdump kernel runs on >=MAX_VIRT_CPUS vcpu)
	 */
	if (__this_cpu_read(xen_vcpu_nr(0)) == NULL)
		return;

	...

-boris