Subject: Re: [PATCH] cpu/hotplug: Allow the CPU in CPU_UP_PREPARE state to be
 brought up again.
To:     Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
        "Longpeng (Mike, Cloud Infrastructure Service Product Dept.)" 
        <longpeng2@huawei.com>
Cc:     linux-kernel@vger.kernel.org,
        "Gonglei (Arei)" <arei.gonglei@huawei.com>, x86@kernel.org,
        xen-devel@lists.xenproject.org,
        Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@kernel.org>,
        Valentin Schneider <valentin.schneider@arm.com>,
        Boris Ostrovsky <boris.ostrovsky@oracle.com>,
        Juergen Gross <jgross@suse.com>,
        Stefano Stabellini <sstabellini@kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        "H. Peter Anvin" <hpa@zytor.com>
References: <20211122154714.xaoxok3fpk5bgznz@linutronix.de>
From:   Dongli Zhang <dongli.zhang@oracle.com>
Message-ID: <d5f43af4-f91f-ef9f-9c33-8d349f59b23c@oracle.com>
Date:   Tue, 23 Nov 2021 13:21:48 -0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.6.1
In-Reply-To: <20211122154714.xaoxok3fpk5bgznz@linutronix.de>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Vk1RU3BESklSS3lGZkk0SlRuWUhEc2RxdG1Hb3puanIzSzg2QnBXeUEzcnc5?=
 =?utf-8?B?M0FPYTZqejJyS08raTBhVDA2c240Q0VJSzAxNUtIaWtBdTFBbUJraGx3aFhx?=
 =?utf-8?B?TnlzVDEza2dmQUNab0ZuYWJxRDBDaXZXTFJkT2JJMzhwSDN6MzYwTEthRmZL?=
 =?utf-8?B?dDJUb293L2RzUlhzRFNzRWRGcVpJY0Z6Z3RpTFhOSVYwWUI4ZWF3VzFEZkN4?=
 =?utf-8?B?aThBTjE5cGFrODB1STNjaWVENGwyWktpZG5BbjF3ckJRY1R6ZS90L0ttY01K?=
 =?utf-8?B?VTNPWnVOZktFQzZ2OFVhMkxlK1RrWS9VWElYdmxuZXpNOWI1N08zODExTUxv?=
 =?utf-8?B?RnFtVncwcEtXekhCbXZNYUhkb1FuZVhKZEdRK20wTWZPeDN1OHZKZThJNjAr?=
 =?utf-8?B?TVVSTjNJOU05QXlIZFd4eVJMbkl3ekVVUGhtVDN0SUpvZmJtcEFXcTdRb0hi?=
 =?utf-8?B?YXoyTWlIMWJLRnZuQU9Sd2VhZlBsVmdvcmsrU2J6UjRiRkRyalRnZlFCZTk1?=
 =?utf-8?B?L0hoeUh6bnE0UHNWWUF4LzNiZDNoOUpKRjF5NHgzQlU0R0NYVHRzQ1pTd1lI?=
 =?utf-8?B?T3JhdWZRbmU5MzdETUZDcTBNUWI1eHJoTGdmYWhkZXBBSHVhcE02TFRqNU9N?=
 =?utf-8?B?Z2lSZzVkMDNGME01b0dTSU9WQnNUV05tN1gvWGZxc3dweFB3MTBnZTdWWU5E?=
 =?utf-8?B?bFBNTmJZdk1BcXRmUEtteVBHWUhIR0NqNHBlQ1pUT0RFd0N4K3ByWXlvRzcw?=
 =?utf-8?B?TEc1WFhNYTdRR2RXMGpwcW56RTVKZzMzU3U0cTZPL3VhbkdDbFhMMW5ZR052?=
 =?utf-8?B?MHpEV1BXcy9CNUZrcDFqenQ1TWZ4N2gyR2hRR3JEbnV1NVRsQWtIS3gwT01N?=
 =?utf-8?B?ckRWNnNjY2xPUzJnWXVtdHMrZ3pKcTZGaVJGTldqNUlaYllwYm9xdmFDNGpt?=
 =?utf-8?B?UHpDYWdDM1VvRWJWcGRRS1ZTWW5pclRpWGhiY2hWYjR3WHYwSlRIY3ZUQkZy?=
 =?utf-8?B?aVA2Rk9MM0pVSUpheVNLV3dFQVZORktBMWdWbXppZ01aT1k2WmoyeGl6eHUz?=
 =?utf-8?B?OVQvNW1LYzdHdjVuWGFNRXlFWDFIVXZGbDlLbFlKR3laQ01IYkZ3SFVrVGgz?=
 =?utf-8?B?a3lsS3hBOVk5ZlNGaVlzN296bXY2SndrRmovZkNRN21ZNFNVVHpGbnpJcWE2?=
 =?utf-8?B?RDVDVjN5bTBORGxVQVF5d3c4TGNqQ2wzT2tVb0F2YzdKdlhaMnRmODBQR1Jl?=
 =?utf-8?B?UFVXS2ZBMG83MGtwbE9yUDltTTB2QkNnUHhFWjM4QVZvN3phSGRLaGhpTisx?=
 =?utf-8?B?eDNoMTdHWXY3YWpQbTl0NEdrb1NlYThPajViWkJSR0p2S0c1WitYVFF2ckhp?=
 =?utf-8?B?OUlDNEtoUXRaS3h3QVNuZ3J4VTBYVm5BMGhkbTgwTkVlMm95YVl5MnhvbDJP?=
 =?utf-8?B?M1dYV2JpZStBY2pEM2Z5dHh3Ym5ubGJaSnhVZHJpeVY4aXBVQXVkcVRwYkdv?=
 =?utf-8?B?OHU4dWlua094QXR0RXdmYVJibWVQRG9JMUZydGdrelFlUGFqSjk2anExMXRj?=
 =?utf-8?B?ZW5TOGlGdlNnREVEbUZ3T0dvU2hQOXZmdytucFJxN1NpUnpqYW1MK0RHSHd5?=
 =?utf-8?B?ZnZHYUZUWFFwZFFCR1kyM0xEZGRLa2ZmVytaeUpxenhIMzhMcCtoUFdoZWxM?=
 =?utf-8?B?STZpb1ZhS1hvNlh3RDc5V0lvMXN0V0o1eEtkOUxwSEtDN1lHRjFzNGp4bFM5?=
 =?utf-8?B?RG5GU2JrcjRiTW9JQTFCdGhZR2pKcGM0Qm5oOFRGRlh0dHo3NmJLUXBXMDVw?=
 =?utf-8?B?Qkc5YXJNa05yOXIrbGpUa2kyT1JIQ3llTkw1VHVQTkVpN2krWGdxWUc4cFJ3?=
 =?utf-8?B?cmltT0JtbW9FUkdWYXNEa01FT2s4Y2JKQW0zOFM2YWNHUjkyejJJaUNJZ3oy?=
 =?utf-8?B?bkZFL0pId1BLQ1YzSkM5eHVvZkJIQ1pNcWRYKzZRSytNOXB3ajBxaGZkQVNq?=
 =?utf-8?B?VGd6VXJ1U1RoUlJrTWxDRWp2TEpmY1lOQUlZWk5pUFhKMk5nNVNtQk5udDcv?=
 =?utf-8?B?YnM0TlB6RUVPZ2ZOaE54RXZvL251WkN0dVVYR21acVp6Q3hGNmtqSUx1SytX?=
 =?utf-8?B?eE1jcDNIZ1o5M09CeXV3ZXdoUDJKYU96OGgrdm1FUy9ScE9vR1d2UHpRY3pK?=
 =?utf-8?B?UFNoeGJLc3hoODdFMGZMOFUzMTdndUJkbWZyclFxTlU2dFV4Z2ZneFM5a3Az?=
 =?utf-8?B?d2dHZGd2UklCcGJZU2FnUmx0VUZnPT0=?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c5bc3b28-98b1-49dc-b262-08d9aec744ae
X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2663.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Nov 2021 21:21:52.9576
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: CKw89JwUu6u3pGylkQF1WjRRY5DQErLoI4RhG8zYH+wmUKXpMfr92CxJvufLHm2/yh4WPRDAf5ay0/AdTjNpZQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB3809
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10177 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 spamscore=0
 bulkscore=0 suspectscore=0 mlxscore=0 mlxlogscore=999 phishscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000
 definitions=main-2111230103
X-Proofpoint-GUID: sp0GUbpNiPEKC7yCNsMBV4Jbs65_oaRY
X-Proofpoint-ORIG-GUID: sp0GUbpNiPEKC7yCNsMBV4Jbs65_oaRY
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Tested-by: Dongli Zhang <dongli.zhang@oracle.com>


The bug fixed by commit 53fafdbb8b21 ("KVM: x86: switch KVMCLOCK base to
monotonic raw clock") may leave the cpu_hotplug_state at CPU_UP_PREPARE. As a
result, to online this CPU again (even after removal) is always failed.

I have tested that this patch works well to workaround the issue, by introducing
either a mdeley(11000) or while(1); to start_secondary(). That is, to online the
same CPU again is successful even after initial do_boot_cpu() failure.

1. add mdelay(11000) or while(1); to the start_secondary().

2. to online CPU is failed at do_boot_cpu().

3. to online CPU again is failed without this patch.

# echo 1 > /sys/devices/system/cpu/cpu4/online
-su: echo: write error: Input/output error

4. to online CPU again is successful with this patch.

Thank you very much!

Dongli Zhang

On 11/22/21 7:47 AM, Sebastian Andrzej Siewior wrote:
> From: "Longpeng(Mike)" <longpeng2@huawei.com>
> 
> A CPU will not show up in virtualized environment which includes an
> Enclave. The VM splits its resources into a primary VM and a Enclave
> VM. While the Enclave is active, the hypervisor will ignore all requests
> to bring up a CPU and this CPU will remain in CPU_UP_PREPARE state.
> The kernel will wait up to ten seconds for CPU to show up
> (do_boot_cpu()) and then rollback the hotplug state back to
> CPUHP_OFFLINE leaving the CPU state in CPU_UP_PREPARE. The CPU state is
> set back to CPUHP_TEARDOWN_CPU during the CPU_POST_DEAD stage.
> 
> After the Enclave VM terminates, the primary VM can bring up the CPU
> again.
> 
> Allow to bring up the CPU if it is in the CPU_UP_PREPARE state.
> 
> [bigeasy: Rewrite commit description.]
> 
> Signed-off-by: Longpeng(Mike) <longpeng2@huawei.com>
> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> Link: https://urldefense.com/v3/__https://lore.kernel.org/r/20210901051143.2752-1-longpeng2@huawei.com__;!!ACWV5N9M2RV99hQ!d4sCCXMQV7ekFwpd21vo1_9K-m5h4VZ-gE8Z62PLL58DT4VJ6StH57TR_KpBdbwhBE0$ 
> ---
> 
> For XEN: this changes the behaviour as it allows to invoke
> cpu_initialize_context() again should it have have earlier. I *think*
> this is okay and would to bring up the CPU again should the memory
> allocation in cpu_initialize_context() fail.
> 
>  kernel/smpboot.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/kernel/smpboot.c b/kernel/smpboot.c
> index f6bc0bc8a2aab..34958d7fe2c1c 100644
> --- a/kernel/smpboot.c
> +++ b/kernel/smpboot.c
> @@ -392,6 +392,13 @@ int cpu_check_up_prepare(int cpu)
>  		 */
>  		return -EAGAIN;
>  
> +	case CPU_UP_PREPARE:
> +		/*
> +		 * Timeout while waiting for the CPU to show up. Allow to try
> +		 * again later.
> +		 */
> +		return 0;
> +
>  	default:
>  
>  		/* Should not happen.  Famous last words. */
>