Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [PATCH v2 1/1] xen: delay xen_hvm_init_time_ops() if kdump is
 boot on vcpu>=32
To:     Boris Ostrovsky <boris.ostrovsky@oracle.com>,
        xen-devel@lists.xenproject.org, x86@kernel.org
Cc:     jgross@suse.com, sstabellini@kernel.org, tglx@linutronix.de,
        mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com,
        hpa@zytor.com, joe.jin@oracle.com, linux-kernel@vger.kernel.org
References: <20211028012543.8776-1-dongli.zhang@oracle.com>
 <1f19674f-e608-1faa-5656-fec853297198@oracle.com>
From:   Dongli Zhang <dongli.zhang@oracle.com>
Message-ID: <fe6d7aaf-97b5-dfa2-75e5-1a072f4aac92@oracle.com>
Date:   Thu, 4 Nov 2021 10:59:55 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.6.1
In-Reply-To: <1f19674f-e608-1faa-5656-fec853297198@oracle.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cG9oNmxUVk1OclBNZW10WUl4eDVJN0tsSUpLalVyd0JxU0UvOFNkR3FWQVh2?=
 =?utf-8?B?bXBteWliMmdGcnhyL24vMWg2R0VpcXJsK0JLUjBZS3VvcDhRck5aZmFLTnY1?=
 =?utf-8?B?NG5jSjJIcHJuUUNNYjRmeVpzVXV6Tnk5ck9GMEZ0M1cxb1VWS29CVXFzSEZw?=
 =?utf-8?B?WEdKYmx3djM2MS9GV3hscWdzNjc4YlR6ME0wL09xV1BTWkpVY2EzZ2V3Z3J0?=
 =?utf-8?B?YXZFbmx1S0UyRG9CTEpPTUc2Mm5NYWFXcWE1R3drQ2ZLZTVaQWh3S3pkSllT?=
 =?utf-8?B?SlBVRE9sOHJWdnY0TytnUHRUVXg0VWx2VDczcmtUd29YV3Q0eC9ZcDZZM0Vq?=
 =?utf-8?B?dVoxUEJOd3lyRU9GZlM0TGd1NS82NkNISHh3VFRTYUw4UFRYM0s4NXdzNkhN?=
 =?utf-8?B?QlU4THJSVFZPUjJsaE53Lzc2MG90SVFMdElrdEhtZ05acTV3Z1hURmU1ck5a?=
 =?utf-8?B?eVlWT01xalZqVzJCZzY1NzlVWkhhV0FtWmxIQ0t1YVRyQzhsdlhDbCtYUW5l?=
 =?utf-8?B?WW12eS9NVis3MzFCeGRsQzlUWjg2aThNQkdqYmszSkZCMlJmb1YvbER4ZnZa?=
 =?utf-8?B?ek9Dcnp6UHNISktZdFlNUE9sNWNhWERpN0p1U3VHMDI5TGM4SjVTWjBKWVpj?=
 =?utf-8?B?bTFqNzNGV3pRK3RNeU05VFowWFhhNEIvcEt0SnBxZmZWemZlSVhQQWVia0cx?=
 =?utf-8?B?ckx1K1B0blJkNEdvL2lHdkpDK2RncUYyb3Q3a2x3Smo3QlBXa1VhSFVJVGli?=
 =?utf-8?B?bTUyYjAwZlFVcTZuV2VEekk4UkFwdjV0Q1FxVzhlb1k1K1NkNkQ0SFVhbFdr?=
 =?utf-8?B?ZWJYdDhrMWFRdlNNeFFWSHQ2Rk5rQ2NuTjRxK0RUTWc3RFg4eWlxRVhRU1hq?=
 =?utf-8?B?OFJleEM2N0VyVzJ4cW1pbjZBamd4Y2crd2l5TFNpMTlCSWxBUHZ3b0NHRy83?=
 =?utf-8?B?TTdJdWZ0SU5uaGxiZ3V2cTBnL3N1WXZxZmszN3hHL0M3VWYrTm1VU25wcFZY?=
 =?utf-8?B?Y0QxUVNGT0tsd1BqcXIzOVRBVGhzSW51Rno1VEJML0w0ZElNUjhDdnhVREM1?=
 =?utf-8?B?dXhRMGdhY0V1aUpMdGwrdkg2bnpKOFlaY1kvTDJwcEpCQkUxMmo1eXpDTm9p?=
 =?utf-8?B?NDRHZ0ptSTR4c2g0bkdtSGltRElxTW9OdWZmbmxiVTRzK3ZUTUFNbXlPQ2lH?=
 =?utf-8?B?S3ZMbkhBbnFLY2U0SkdDR2ttc2c4YUd0b0RpUFpmaUIyejlWMHRzM2E0d0Nw?=
 =?utf-8?B?M2c4eUtiUExPc252TXlrT2labTdDS052RHV3SDNua3VCT2JVekQwcnY2bFhT?=
 =?utf-8?B?dzNiZEpqcWVVT2Q2UmV6MmJxelNlb3I2S1BseUtuczVPclhVcEpNeUFNSlVi?=
 =?utf-8?B?RldMenlZM3Bza0g4eXp5Yng1d2phclNjSDE3UVpQeDZqejRHTXZFMGUvN0tD?=
 =?utf-8?B?NHhzLzUxTExMWnhnT3RsdmJYOG1XMHlkcVVJTDFZSFRoZDZIWHdieWk5REhE?=
 =?utf-8?B?NFBoUlZCZE8vdTA2c25IMVpnNWM4eThzUlFQLzdmTWowd2JUdlpGQnJFeGpD?=
 =?utf-8?B?TWJoY1IySmUxTVJad0psV1RlK0dEQjA3SjlHc0FVVEphMDdkbnc4cUxtVkh4?=
 =?utf-8?B?cGgrSFE5OHNaYUg0OHcxNmd6d2xXTHlXZ1Npc3RiYkZHa0ZGQUM4VWcvNmx0?=
 =?utf-8?B?dGFrZllhOWFGSUwyTmxEejlHcGVwb0dZQXhYRXpRZDlyTkRiSWwzazBENlhL?=
 =?utf-8?B?UzhiMlJ5aEhHakdkZXRlVjJoVEFzSitROVh0d2VOYVN2VDZIcWRoYklsbGY3?=
 =?utf-8?B?MkdhUy9RVDhnMVZqZERrb2FTcmI3QkJabVBXMUt3Y2luZWwzVFN6U0F6YjNr?=
 =?utf-8?B?UFE4T1ByWmxNaUhoSVJFQ0FkQSs5andSSVJUcEtVSlU1ZjBmWng1Skt1Y1Uy?=
 =?utf-8?B?a3U5azJwUERKTTc5NEV1UkpqU05yTE5nTkpwdEUwcldSOU82cG9JMkdTOG13?=
 =?utf-8?B?RnowaFErMXZLOXRLMnlsSmtvNGJyWUF0MWwrd0tKRkRjMGNVTENZS2FQSFBu?=
 =?utf-8?B?cERoYjF3ZzBQRWJjY0hsVUVFY1NWMjNEMERsa0V1Z1JJdkQzMVFIUTlDZHVR?=
 =?utf-8?B?SFQ0N05XWnpucEY5ckF5ZlE2U0lHRzVSSzYzRy80b1RLWVdLWFA4NGxtbkxM?=
 =?utf-8?Q?ZYZXBR/ypWAA58uNEev+yt2pPEeCtEX+pKMIcIijc1ba?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2e3d8a68-fd31-440a-0e77-08d99fbce585
X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2663.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Nov 2021 17:59:50.4725
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 7C9LCDFLWzWMBDJa6BtdpG3EoSiudUc0r8wunuTmmFLVqSwBrCEsCxMjaZ+TzxNh+q+X+Bl+5WogKk3KhE0wZg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB5550
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10158 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 phishscore=0 malwarescore=0
 mlxscore=0 suspectscore=0 mlxlogscore=999 adultscore=0 spamscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000
 definitions=main-2111040069
X-Proofpoint-ORIG-GUID: HNmlFvK5jXFawfujbpyNh2aApfUSvMW9
X-Proofpoint-GUID: HNmlFvK5jXFawfujbpyNh2aApfUSvMW9
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Boris,

On 11/1/21 10:34 AM, Boris Ostrovsky wrote:
> 
> On 10/27/21 9:25 PM, Dongli Zhang wrote:
>> The sched_clock() can be used very early since
>> commit 857baa87b642 ("sched/clock: Enable sched clock early"). In addition,
>> with commit 38669ba205d1 ("x86/xen/time: Output xen sched_clock time from
>> 0"), kdump kernel in Xen HVM guest may panic at very early stage when
>> accessing &__this_cpu_read(xen_vcpu)->time as in below:
>>
>> setup_arch()
>>   -> init_hypervisor_platform()
>>       -> x86_init.hyper.init_platform = xen_hvm_guest_init()
>>           -> xen_hvm_init_time_ops()
>>               -> xen_clocksource_read()
>>                   -> src = &__this_cpu_read(xen_vcpu)->time;
>>
>> This is because Xen HVM supports at most MAX_VIRT_CPUS=32 'vcpu_info'
>> embedded inside 'shared_info' during early stage until xen_vcpu_setup() is
>> used to allocate/relocate 'vcpu_info' for boot cpu at arbitrary address.
>>
>> However, when Xen HVM guest panic on vcpu >= 32, since
>> xen_vcpu_info_reset(0) would set per_cpu(xen_vcpu, cpu) = NULL when
>> vcpu >= 32, xen_clocksource_read() on vcpu >= 32 would panic.
>>
>> This patch delays xen_hvm_init_time_ops() to later in
>> xen_hvm_smp_prepare_boot_cpu() after the 'vcpu_info' for boot vcpu is
>> registered when the boot vcpu is >= 32.
>>
>> Another option is to always delay xen_hvm_init_time_ops() for any vcpus
>> (including vcpu=0). Since to delay xen_hvm_init_time_ops() may lead to
>> clock backward issue,
> 
> 
> This is referring to
> https://lists.xenproject.org/archives/html/xen-devel/2021-10/msg01516.html I
> assume?

No.

So far there are clock forward (e.g., from 0 to 65345) issue and clock backward
issue (e.g., from 2.432 to 0).

The clock forward issue can be resolved by above link to enforce clock update
during vcpu info registration. However, so far I am only able to reproduce clock
forward when "taskset -c 33 echo c > /proc/sysrq-trigger".

That is, I am not able to see any clock forward drift during regular boot (on
CPU=0), without the fix at hypervisor side.

The clock backward issue is because the native clock source is used if we delay
the initialization of xen clock source. We will see a backward when the source
is switched from native to xen.

> 
> 
>>   it is preferred to avoid that for regular boot (The
>> pv_sched_clock=native_sched_clock() is used at the very beginning until
>> xen_sched_clock() is registered). That requires to adjust
>> xen_sched_clock_offset. That's why we only delay xen_hvm_init_time_ops()
>> for vcpu>=32.
> 
> 
> We delay only on VCPU>=32 because we want to avoid the clock going backwards due
> to hypervisor problem pointed to be the link above, not because we need to
> adjust xen_sched_clock_offset (which we could if we wanted).

I will add that.

Just to clarify that so far I am not able to reproduce the clock backward issue
during regular boot (on CPU=0), when the fix is not available at hypervisor side.

> 
> 
>>
>> This issue can be reproduced on purpose via below command at the guest
>> side when kdump/kexec is enabled:
>>
>> "taskset -c 33 echo c > /proc/sysrq-trigger"
>>
>> Reference:
>> https://lists.xenproject.org/archives/html/xen-devel/2021-10/msg00571.html
>> Cc: Joe Jin <joe.jin@oracle.com>
>> Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
>> ---
>> Changed since v1:
>>    - Add commit message to explain why xen_hvm_init_time_ops() is delayed
>>      for any vcpus. (Suggested by Boris Ostrovsky)
>>    - Add a comment in xen_hvm_smp_prepare_boot_cpu() referencing the related
>>      code in xen_hvm_guest_init(). (suggested by Juergen Gross)
>>
>>   arch/x86/xen/enlighten_hvm.c | 20 +++++++++++++++++++-
>>   arch/x86/xen/smp_hvm.c       |  8 ++++++++
>>   2 files changed, 27 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/x86/xen/enlighten_hvm.c b/arch/x86/xen/enlighten_hvm.c
>> index e68ea5f4ad1c..7734dec52794 100644
>> --- a/arch/x86/xen/enlighten_hvm.c
>> +++ b/arch/x86/xen/enlighten_hvm.c
>> @@ -216,7 +216,25 @@ static void __init xen_hvm_guest_init(void)
>>       WARN_ON(xen_cpuhp_setup(xen_cpu_up_prepare_hvm, xen_cpu_dead_hvm));
>>       xen_unplug_emulated_devices();
>>       x86_init.irqs.intr_init = xen_init_IRQ;
>> -    xen_hvm_init_time_ops();
>> +
>> +    /*
>> +     * Only MAX_VIRT_CPUS 'vcpu_info' are embedded inside 'shared_info'
>> +     * and the VM would use them until xen_vcpu_setup() is used to
>> +     * allocate/relocate them at arbitrary address.
>> +     *
>> +     * However, when Xen HVM guest panic on vcpu >= MAX_VIRT_CPUS,
>> +     * per_cpu(xen_vcpu, cpu) is still NULL at this stage. To access
>> +     * per_cpu(xen_vcpu, cpu) via xen_clocksource_read() would panic.
>> +     *
>> +     * Therefore we delay xen_hvm_init_time_ops() to
>> +     * xen_hvm_smp_prepare_boot_cpu() when boot vcpu is >= MAX_VIRT_CPUS.
>> +     */
>> +    if (xen_vcpu_nr(0) >= MAX_VIRT_CPUS)
>> +        pr_info("Delay xen_hvm_init_time_ops() as kernel is running on
>> vcpu=%d\n",
>> +            xen_vcpu_nr(0));
>> +    else
>> +        xen_hvm_init_time_ops();
>> +
>>       xen_hvm_init_mmu_ops();
>>     #ifdef CONFIG_KEXEC_CORE
>> diff --git a/arch/x86/xen/smp_hvm.c b/arch/x86/xen/smp_hvm.c
>> index 6ff3c887e0b9..f99043df8bb5 100644
>> --- a/arch/x86/xen/smp_hvm.c
>> +++ b/arch/x86/xen/smp_hvm.c
>> @@ -19,6 +19,14 @@ static void __init xen_hvm_smp_prepare_boot_cpu(void)
>>        */
>>       xen_vcpu_setup(0);
>>   +    /*
>> +     * The xen_hvm_init_time_ops() is delayed from
>> +     * xen_hvm_guest_init() to here to avoid panic when the kernel
>> +     * boots from vcpu>=MAX_VIRT_CPUS (32).
>> +     */
> 
> 
> How about
> 
>   /* Deferred call to xen_hvm_init_time_ops(). See comment in
> xen_hvm_guest_init() */
> 

I will do that.

Thank you very much!

Dongli Zhang

> 
> -boris
> 
> 
> 
>> +    if (xen_vcpu_nr(0) >= MAX_VIRT_CPUS)
>> +        xen_hvm_init_time_ops();
>> +
>>       /*
>>        * The alternative logic (which patches the unlock/lock) runs before
>>        * the smp bootup up code is activated. Hence we need to set this up