Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [PATCH 0/2] Fix the Xen HVM kdump/kexec boot panic issue
To:     Juergen Gross <jgross@suse.com>, xen-devel@lists.xenproject.org
Cc:     linux-kernel@vger.kernel.org, x86@kernel.org,
        boris.ostrovsky@oracle.com, sstabellini@kernel.org,
        tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com,
        andrew.cooper3@citrix.com, george.dunlap@citrix.com,
        iwj@xenproject.org, jbeulich@suse.com, julien@xen.org, wl@xen.org,
        joe.jin@oracle.com
References: <20211012072428.2569-1-dongli.zhang@oracle.com>
 <cb2e1a63-de79-5050-bf2b-bfb302997381@suse.com>
From:   Dongli Zhang <dongli.zhang@oracle.com>
Message-ID: <2794895b-6417-7164-8417-0f1edc52ae10@oracle.com>
Date:   Tue, 12 Oct 2021 08:50:17 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.6.1
In-Reply-To: <cb2e1a63-de79-5050-bf2b-bfb302997381@suse.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?V1NxTkxSYW1iVWJQQnpwNk1YYjNFYlQyWWZIblR0U0tmNFJEMXFPUm5wY2dq?=
 =?utf-8?B?R2l5T1lLYWNhYlA1SXhHait2Z3Bqbk4zakdTakpkUVpXaWJsZ2plOXhxVzlP?=
 =?utf-8?B?bWxaU3AyQ2N5emxXU3ZoMFc3eDBaelJobGFBM0FSVDI1Ly9ob0tSL21ScDR5?=
 =?utf-8?B?NXVuMWtFc0hqczJpcDJzYVhFVUUvVmNQcEdJUUQ2TFBESld0M3FSUGlGeCs4?=
 =?utf-8?B?V0pqak90TWxxUzYwUUJkSmxGZFFyWkQ2SUNDT3lPWE9iY2RoN3B0b2llV0JM?=
 =?utf-8?B?MkFMV28vd0ZVMExtbmprNytqcU1LRWJyUHhIYlhDQWdVZ2pPRUNNNXVCdmxk?=
 =?utf-8?B?NTFYS0lrbngzNCtKUjlZSnl5ZSsvYjRZeE8vbzJKc08xNGxEV0ZPZWtxc1ZD?=
 =?utf-8?B?QWR3ZlNNSkRwSmZnZkFGejViLyt1NkxQcnE4RUlDSHJQRGZzcVhSaXRaRkFS?=
 =?utf-8?B?VUdUeVhxMkZJd0xuM05mK1RrQUsrSnZGeXMrOVNoTlIxYVpXU01GRld0WmFK?=
 =?utf-8?B?c2M2UGRGNW92U2hsa0FkdVQ2eDFHdHBDd05vWGxKV1BCZ09uQjVvb0Zwd21C?=
 =?utf-8?B?Z2gzbktOTWR0UDZrdTIxODdGeDk5SkFIaEdJeWkvSnBvWlMxL3BpQ1dsUi9r?=
 =?utf-8?B?S0hQckZ6eEhQVENvNnZVeTAvQ1ozbmk0cWlEUXJiTVB0OXgyYlhWMkUxcVFE?=
 =?utf-8?B?aXYzWUJaeTc1em9RTlFqQlBUY0hlbXRKQWY5SUhuc1JvNWNqVllVVDE2QnFS?=
 =?utf-8?B?L3NFUmorT3lOTWhobXAxVUpYbUYyUmJpZzZPSFp3VE5GZmt6emozUHU5c0lU?=
 =?utf-8?B?UEFDVlVHWWZRN3B1eXVpaEIxUFZOZUhIK2NBWTBWeWtaUWR4OGZ6ZlRma2xK?=
 =?utf-8?B?cTVkVEhhNkRwenlSSVRPSjM4L01jSWp4UndPc3pUbW5KaDdtdXdJZ2V2L01h?=
 =?utf-8?B?dEtDYXRKaXF6M0FtM09hb3hNTkQ4YXZLVSs5U3k3OG5obVRCRC9uS2RKS3dT?=
 =?utf-8?B?Z3cxWVBLL3M2STJWaUxUN0loTmIxSndMKzBEZ0wvNGFHRk9YcjRFNi8waVAy?=
 =?utf-8?B?QmZnS2hKL1JWb09QYW5PamtkandZcnQvRUNwbUI5UUhVbGt4YkY4ajVDNU5M?=
 =?utf-8?B?clJuTGZTc0dMbnZZTms2LzRxVEFFL0FWY0lsdkMwOGo1WmlzWHFBc083RzQ2?=
 =?utf-8?B?VGUvUSs3OGxPVnZMeUZYbHExOENSZkp6K0Z4NnNsaitLa2Y4Z0l0WmNEbzJ5?=
 =?utf-8?B?MjVrOFZiUk40WXhHR0NmcFN4bG1vdFNLVnRuWjY1Lys0bjJHVFVBOW5mV3hF?=
 =?utf-8?B?N0hpZ3kvU1VMaTdleFl6RXdoZzlaWUFoTFZEODd6VFNBZXQ0eUltYU9XYVRs?=
 =?utf-8?B?cXRxcDBQb0lnVy9qRU5yZEVpUUJXZXRCSDBMSzUvY1FpV3o0M2p6eWpHcDVZ?=
 =?utf-8?B?WlQ0aFM1RUpGZlliQUlEbnlId3FXbk9FR3dIL2dGZWdDRWIvSjdIWDAwUWpu?=
 =?utf-8?B?eVhHMGE2OUEwTXNxN0pGVXZDM0hYcEUzbk8zNVJaUGFScEQybUxTL3Q2ZW1i?=
 =?utf-8?B?Z3Vud2M5VmE2TEJFcnR6SDJwbiswcHpnUU5SOEp2OFI0eGxsZWFGcC9Xbmtt?=
 =?utf-8?B?RFk3TTFWV2Y3NTc0c2pUcWFCZDhUano2WStxSEpocVRQdWJZQ1NtMEROa1F2?=
 =?utf-8?B?VXh3TktkTXdRejU0L1FwTnE2U250dWYrTnp3Z2Y1cmFRNys2L3FLb2I3YTBH?=
 =?utf-8?B?SU5RU3g3eVEvWlhIVTNSTFNFR1FyVEg2dzhEc2dpank2N1ZHeENLUndlY08r?=
 =?utf-8?Q?oLx2X3Iaoe1y+GQc1TzZepnvLeTUErlSPaTEs=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f9e7cf76-097e-473e-31ab-08d98d97ff46
X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2663.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Oct 2021 15:50:21.4742
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 3rZwxwpPegd7Em1Yb52/IYzQeeyPOfzULvNsdO/uLXzk/m5xXwRyfPZPshC+Df7iU59B3qAdNJIcds1qe4rtBg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB5802
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10135 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 phishscore=0
 suspectscore=0 mlxlogscore=999 bulkscore=0 mlxscore=0 malwarescore=0
 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2109230001 definitions=main-2110120088
X-Proofpoint-GUID: 9ar5uwnedwp_u-K97u1JLQMijareiRop
X-Proofpoint-ORIG-GUID: 9ar5uwnedwp_u-K97u1JLQMijareiRop
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Juergen,

On 10/12/21 1:47 AM, Juergen Gross wrote:
> On 12.10.21 09:24, Dongli Zhang wrote:
>> When the kdump/kexec is enabled at HVM VM side, to panic kernel will trap
>> to xen side with reason=soft_reset. As a result, the xen will reboot the VM
>> with the kdump kernel.
>>
>> Unfortunately, when the VM is panic with below command line ...
>>
>> "taskset -c 33 echo c > /proc/sysrq-trigger"
>>
>> ... the kdump kernel is panic at early stage ...
>>
>> PANIC: early exception 0x0e IP 10:ffffffffa8c66876 error 0 cr2 0x20
>> [    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.15.0-rc5xen #1
>> [    0.000000] Hardware name: Xen HVM domU
>> [    0.000000] RIP: 0010:pvclock_clocksource_read+0x6/0xb0
>> ... ...
>> [    0.000000] RSP: 0000:ffffffffaa203e20 EFLAGS: 00010082 ORIG_RAX:
>> 0000000000000000
>> [    0.000000] RAX: 0000000000000003 RBX: 0000000000010000 RCX: 00000000ffffdfff
>> [    0.000000] RDX: 0000000000000003 RSI: 00000000ffffdfff RDI: 0000000000000020
>> [    0.000000] RBP: 0000000000011000 R08: 0000000000000000 R09: 0000000000000001
>> [    0.000000] R10: ffffffffaa203e00 R11: ffffffffaa203c70 R12: 0000000040000004
>> [    0.000000] R13: ffffffffaa203e5c R14: ffffffffaa203e58 R15: 0000000000000000
>> [    0.000000] FS:  0000000000000000(0000) GS:ffffffffaa95e000(0000)
>> knlGS:0000000000000000
>> [    0.000000] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [    0.000000] CR2: 0000000000000020 CR3: 00000000ec9e0000 CR4: 00000000000406a0
>> [    0.000000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> [    0.000000] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>> [    0.000000] Call Trace:
>> [    0.000000]  ? xen_init_time_common+0x11/0x55
>> [    0.000000]  ? xen_hvm_init_time_ops+0x23/0x45
>> [    0.000000]  ? xen_hvm_guest_init+0x214/0x251
>> [    0.000000]  ? 0xffffffffa8c00000
>> [    0.000000]  ? setup_arch+0x440/0xbd6
>> [    0.000000]  ? start_kernel+0x6a/0x689
>> [    0.000000]  ? secondary_startup_64_no_verify+0xc2/0xcb
>>
>> This is because Xen HVM supports at most MAX_VIRT_CPUS=32 'vcpu_info'
>> embedded inside 'shared_info' during early stage until xen_vcpu_setup() is
>> used to allocate/relocate 'vcpu_info' for boot cpu at arbitrary address.
>>
>>
>> The 1st patch is to fix the issue at VM kernel side. However, we may
>> observe clock drift at VM side due to the issue at xen hypervisor side.
>> This is because the pv vcpu_time_info is not updated when
>> VCPUOP_register_vcpu_info.
>>
>> The 2nd patch is to force_update_vcpu_system_time() at xen side when
>> VCPUOP_register_vcpu_info, to avoid the VM clock drift during kdump kernel
>> boot.
> 
> Please don't mix patches for multiple projects in one series.
> 
> In cases like this it is fine to mention the other project's patch
> verbally instead.
> 

I will split the patchset in v2 and email to different projects.

The core ideas of this combined patchset are:

1. Fix at HVM domU side (kdump kernel panic)

2. Fix at Xen hypervisor side (clock drift issue in kdump kernel)

3. To report (or seek for help) that soft_reset does not work with mainline-xen
so that I am not able to test my patchset with the most recent mainline xen.

Thank you very much!

Dongli Zhang