Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [PATCH linux 1/2] xen: delay xen_hvm_init_time_ops() if kdump is
 boot on vcpu>=32
To:     Boris Ostrovsky <boris.ostrovsky@oracle.com>,
        xen-devel@lists.xenproject.org
Cc:     linux-kernel@vger.kernel.org, x86@kernel.org, jgross@suse.com,
        sstabellini@kernel.org, tglx@linutronix.de, mingo@redhat.com,
        bp@alien8.de, hpa@zytor.com, andrew.cooper3@citrix.com,
        george.dunlap@citrix.com, iwj@xenproject.org, jbeulich@suse.com,
        julien@xen.org, wl@xen.org, joe.jin@oracle.com
References: <20211012072428.2569-1-dongli.zhang@oracle.com>
 <20211012072428.2569-2-dongli.zhang@oracle.com>
 <4faf461f-49ca-5f2b-a7dc-e2b47b904527@oracle.com>
From:   Dongli Zhang <dongli.zhang@oracle.com>
Message-ID: <e4727868-cce2-d705-c84f-1d66204c7814@oracle.com>
Date:   Sun, 24 Oct 2021 22:20:34 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.6.1
In-Reply-To: <4faf461f-49ca-5f2b-a7dc-e2b47b904527@oracle.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dnNZaDQzV0J0Y0Mvc1Awc1JxVXl2YXgxWSs2RzVDQXVOdkVXaUgzQkNZaFNr?=
 =?utf-8?B?aWNLMWsrc2hMWHg5T2o1RFlIdlJkYU9tWUducUNITkdFMzZkdlNBQ1RVUDE1?=
 =?utf-8?B?R0E1VmwxRGxiS1RJYzVRb0NvQzdjb2R2MGc0QXpXV29NRWY2R1pJSTl4SExE?=
 =?utf-8?B?MDE4b2R1MnhIQ3BPS29ydXJuV1RUTUNUSXFuYXZScTdDM1dWOCtvY1Q2NlNl?=
 =?utf-8?B?QTRYUkRNSW85TU1tclNCeTB5cElkYXloOFlIa0p1TVlUUEJUektYNVBnalMz?=
 =?utf-8?B?RWl4cXNOQ1Z6ZmFnM3hSeVF3c0dpRUlnMHVRcjk2U0lqUzBkWTBJM2JrRjJl?=
 =?utf-8?B?MnUzU0NBQUxhb3JCQVJEWnlRSEw1TkdsL05tQWxFSXJXUmVpR1RIK1pOWFNR?=
 =?utf-8?B?WmpZNithM2xwYUZoOWpucUhBRGVUK1NsQWRNM1JlTFM3M0JIU2JPOC9BVk9l?=
 =?utf-8?B?Rk8zNzk2NExqdHlSSVhkemM0b0pxSDNhVzF1T0RrZkpFamozQi8zYlZqcWo2?=
 =?utf-8?B?Ym1hR0xtQWFDUUNtbUJ5S3dtT1h1RzNqR05rNHhraU1BRGU1eGRubGN6MTQv?=
 =?utf-8?B?T1F4aWhQakhjd1BJQ1ZISVJyM2lkRzdvM25VSTJjMGZIL3d5UHJDemovbk1u?=
 =?utf-8?B?dHY4TDBOc21TWlVuUnJ4eWNqV2o1bzVFWWRMOHlibmlWYkx2K1RudUlxRnVt?=
 =?utf-8?B?dlFjQStLK0dRZnRmY0lESXJYamJUUTBCUGx5aU9tbGFPR1J5cHdmUzZvUTJl?=
 =?utf-8?B?STJkU0JUSTJTOWRXbTRLZHFSS2p1NFBReWY2VVFMZVpoTkN5czNJazNyQnZD?=
 =?utf-8?B?cXlybUpaaHJzcDNieURaWEsxTkc0R1YzdWEzVTdqZ0dSeXErd083OFdpZHFa?=
 =?utf-8?B?SDl3ZHViSU11eE1YTno4enp2Yy9TOGlhYlRLY0xUL3VZQjVEeVJpTkpMYTJj?=
 =?utf-8?B?TjBXRXZUOVJDTjFFK0JxQTJjYWtYVHMvb2dsaVhaWkhlSWZVSnh3NStBMGlQ?=
 =?utf-8?B?bXl6dTlRVzB3K3dxamh0T3M3N084N01WMGJJSjhrZ3RKdkJ6VmI2WVIxQy9K?=
 =?utf-8?B?Q0liaWlHU3NmT044NmtHdW0zc1pwRTZjU2p0aGt5SHk2YmVzU1FERDVtVVdE?=
 =?utf-8?B?bUtndStybDNpUnoxREtJL1dPaXVxZi9ob1B4VWUxa0c2Vnk2bm9ZRGxYS3Rt?=
 =?utf-8?B?djlwYVpDNGJBeDdSK1VWckdsMnlOQWpnejJpaWtCUUFjQWNrT2p0Y2ZqWGZF?=
 =?utf-8?B?ekRjQkZZNFMxcGpoOWV4akh4Yk5rOUN4cXY0MDNvMDRxRjNkWldSQjdObTBa?=
 =?utf-8?B?bXpscGpKSUxSakw2ekpEYktYeGpRc2RTcHByMTdiSHVJZ1FEL2RqQW16ZDZ5?=
 =?utf-8?B?L0dENmwyOUxIQ0ZYUTJ4RWtRVVhLWHBnZmk1cXZuSGpRRHpFZ1AyTy9LRWp6?=
 =?utf-8?B?S1pZc3VKWHFZUktYbUNlcytIU3hzemd6SStrZnpnOXBORkNBc3grU0dZbXRQ?=
 =?utf-8?B?WHF5N3VBNStwTHZxY00vNDRPdjZQYW9BVnRoQW5TVGh3OGppanV3ZWN3WDMv?=
 =?utf-8?B?bFlXaXpNWU9pbnNRS3U1Y1ZhZEZmS0R3TENmdjRpRDh5bUQ1elBHdmNidnZ2?=
 =?utf-8?B?eFNyTlhNQ0tjTXF6d3lWbnQ3dkNIQ0E4YjE3UWdpdmRNTVVjbFNPS3N4bG5S?=
 =?utf-8?B?TUJxTUFubEpiQVpucEtNeWVXMkZyaW11aUVpeVF5TFd3aTJrY01wNmJBRWti?=
 =?utf-8?B?blpqajlhdlhtb3Y0d3lQTzY4cGMvZ29RcFNhTkMrejJtNzNwQkJ1YmxVSFlO?=
 =?utf-8?B?ekpsK0hDQlM0bUhQOFZrVG1IS2todnZxSmptQWQ0dlpMdXFVWWhqK3IwanNC?=
 =?utf-8?B?ZnRncjBWTGtKU2RRR0RiM0hMc3pEbVRRMkFKNS9KN0I1SStxNE9VUWRlcVpY?=
 =?utf-8?B?YzlKK3FoWkN1dEVwZVF5Yi9wWmlHaGl4SUp0WGllaXc4aFhqNnJNZmY0MXRX?=
 =?utf-8?B?WVdnVHFBdzZZblJPVGowWTRZR3VPQ3c1L0gvWlg5U2JMMDlIbEFiQjFhK2d4?=
 =?utf-8?B?TDZEeDU2ZGF4cUw3dURQa0ZXS0RybEcxUTZrczdvSDRrTnQ5aTRVZ1F6NnE0?=
 =?utf-8?B?bFZKVlFsQk8yVVNCVUVoc0JZRFBhNGx5U1l5YlY1VTJDdEVoVytyWkFLbStE?=
 =?utf-8?Q?lqEdKop19Mv4Qh6JdFf4OuwK5WvuftMu8cLELKOuG1Ho?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 69ec7a55-bff4-48c1-7bf2-08d997772ea5
X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2663.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Oct 2021 05:20:39.1843
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: fSaXwrkX8hKQIT78nVwVzB+QS/NwOi3p8XBovCFR3Ryq0AxrTGc2d14KyrwFqfmXgSG05Ql04LsEtt7q7TQ+xQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB4354
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10147 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 phishscore=0 adultscore=0
 suspectscore=0 mlxscore=0 bulkscore=0 mlxlogscore=999 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2109230001
 definitions=main-2110250029
X-Proofpoint-ORIG-GUID: Jv6xc1rSMIM71tnBkZvx1F4rNsixY7bV
X-Proofpoint-GUID: Jv6xc1rSMIM71tnBkZvx1F4rNsixY7bV
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Boris,

On 10/12/21 10:17 AM, Boris Ostrovsky wrote:
> 
> On 10/12/21 3:24 AM, Dongli Zhang wrote:
>> The sched_clock() can be used very early since upstream
>> commit 857baa87b642 ("sched/clock: Enable sched clock early"). In addition,
>> with upstream commit 38669ba205d1 ("x86/xen/time: Output xen sched_clock
>> time from 0"), kdump kernel in Xen HVM guest may panic at very early stage
>> when accessing &__this_cpu_read(xen_vcpu)->time as in below:
> 
> 
> Please drop "upstream". It's always upstream here.
> 
> 
>> +
>> +    /*
>> +     * Only MAX_VIRT_CPUS 'vcpu_info' are embedded inside 'shared_info'
>> +     * and the VM would use them until xen_vcpu_setup() is used to
>> +     * allocate/relocate them at arbitrary address.
>> +     *
>> +     * However, when Xen HVM guest panic on vcpu >= MAX_VIRT_CPUS,
>> +     * per_cpu(xen_vcpu, cpu) is still NULL at this stage. To access
>> +     * per_cpu(xen_vcpu, cpu) via xen_clocksource_read() would panic.
>> +     *
>> +     * Therefore we delay xen_hvm_init_time_ops() to
>> +     * xen_hvm_smp_prepare_boot_cpu() when boot vcpu is >= MAX_VIRT_CPUS.
>> +     */
>> +    if (xen_vcpu_nr(0) >= MAX_VIRT_CPUS)
> 
> 
> What about always deferring this when panicing? Would that work?
> 
> 
> Deciding whether to defer based on cpu number feels a bit awkward.
> 
> 
> -boris
> 

I did some tests and I do not think this works well. I prefer to delay the
initialization only for VCPU >= 32.

This is the syslog if we always delay xen_hvm_init_time_ops(), regardless
whether VCPU >= 32.

[    0.032372] Booting paravirtualized kernel on Xen HVM
[    0.032376] clocksource: refined-jiffies: mask: 0xffffffff max_cycles:
0xffffffff, max_idle_ns: 1910969940391419 ns
[    0.037683] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:64
nr_node_ids:2
[    0.041876] percpu: Embedded 49 pages/cpu s162968 r8192 d29544 u262144

--> There is a clock backwards from 0.041876 to 0.000010.

[    0.000010] Built 2 zonelists, mobility grouping on.  Total pages: 2015744
[    0.000012] Policy zone: Normal
[    0.000014] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.0-rc6xen+
root=UUID=2a5975ab-a059-4697-9aee-7a53ddfeea21 ro text console=ttyS0,115200n8
console=tty1 crashkernel=512M-:192M


This is because the initial pv_sched_clock is native_sched_clock(), and it
switches to xen_sched_clock() in xen_hvm_init_time_ops(). Is it fine to always
have a clock backward for non-kdump kernel?

To avoid the clock backward, we may register a dummy clocksource which always
returns 0, before xen_hvm_init_time_ops(). I do not think this is reasonable.

Thank you very much!

Dongli Zhang