Message-ID: <4d04ea65-ea74-dcfd-5b0e-984b44f91961@oracle.com>
Date:   Wed, 24 Nov 2021 21:17:34 -0500
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.3.1
Subject: Re: [PATCH] cpu/hotplug: Allow the CPU in CPU_UP_PREPARE state to be
 brought up again.
Content-Language: en-US
To:     Thomas Gleixner <tglx@linutronix.de>,
        Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
        "Longpeng (Mike, Cloud Infrastructure Service Product Dept.)" 
        <longpeng2@huawei.com>
Cc:     linux-kernel@vger.kernel.org,
        "Gonglei (Arei)" <arei.gonglei@huawei.com>, x86@kernel.org,
        xen-devel@lists.xenproject.org,
        Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@kernel.org>,
        Valentin Schneider <valentin.schneider@arm.com>,
        Juergen Gross <jgross@suse.com>,
        Stefano Stabellini <sstabellini@kernel.org>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        "H. Peter Anvin" <hpa@zytor.com>
References: <20211122154714.xaoxok3fpk5bgznz@linutronix.de>
 <87y25djhaj.ffs@tglx>
From:   Boris Ostrovsky <boris.ostrovsky@oracle.com>
In-Reply-To: <87y25djhaj.ffs@tglx>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?am0rWXZEd3NSV1ByMTdSSy9SR0VEUTRoRTk3SUR0RzVpN3FhY3JXOCthUE05?=
 =?utf-8?B?ZmJBQkZPRE50RnZPOWhzQm9KMXdOR3habytTV1lyNHJibktKcXZnNHlxYzhz?=
 =?utf-8?B?dFB6U2NFRExyb0J3blJ6a1FBQmZROVRMaHczL1dmSkNxOC81S2tON3VNd3lu?=
 =?utf-8?B?eWgzc0RTN0hSL3kxdGhxaDR0WHA1OGJEUVljVHJJUlp6Qkl4UzUzV2pkbnE3?=
 =?utf-8?B?bkRJUUd2NGY2dEV2U1lXeFJVdEdnN0JIbTJZeE9La0hXYnpaNndxMUhxdVpa?=
 =?utf-8?B?dXh2TTJmaWRhTm8yV3Vnclc1a01YSTBwVmNnREFwcXZkeStaMjRSQldxYzdM?=
 =?utf-8?B?Ky9Rd0xvc0F1VG9tUUNqVGorTk5ZMHNWSVZZUTFCWmZsUmdqeDBuTmgzR1lV?=
 =?utf-8?B?SElDejF2ZTBSYWF3a3lIeWFqRDFmeXpvVkxma2ttSTdEVXNvVTFhaDhJOTNq?=
 =?utf-8?B?NVJvYjZvb3Z0Wk5MWmNCaVhUZXhDTUVvdE1HRzVkMWtPdHZJZHNIOHlsMXhC?=
 =?utf-8?B?T2xyMHpxZndEZHFJellQUmhpTzkrQUVhaTdEQkxmd2dLZ2diOUp6ZlRUK1FF?=
 =?utf-8?B?YkVCYXQ0WmQ1S2dZeUwzODdnZStLeDZNcE8wRUV5YkU2VjRaRkFIWDMvbzRN?=
 =?utf-8?B?S1VIQUc4Uyt3ckVSRmN1WHhjek5QTFNqWTF5QzZLN0F1OEJTd1FVR2pidmxz?=
 =?utf-8?B?WGQ3NUlOMnlPMmxIVkpZS1l6bGdQeUVJNVUzVERHdTh2UGIvUHQxM1QzV0pz?=
 =?utf-8?B?ZnczdWZneVF1Tlh5bnIvRUNNTld4Q3NESnZsMFpRcTNxL1BJVDZXOWR3Zk40?=
 =?utf-8?B?OGRtWmVHOEhLbmxDY2hwbTR3czVuVTJJQTBRc1lBcUNGRWEwSE5OaVRyRmNz?=
 =?utf-8?B?b21Qd2wvblBKVWtoTkc0MGEvaHhuNCsyR1pSZ0NxRUZ6WkxpQldOSEtkcTd1?=
 =?utf-8?B?Nmd3T0EvZjhUVzZPMzByem5HMkFleFRZdG1OcmpvRVVHV0hkb01GSldLQXlU?=
 =?utf-8?B?Y3Nnd3RKUXNnTXhkNm8zRzhlUXIxWVIyZm51aFJseHVob3lpTW9vblY1L3o2?=
 =?utf-8?B?ejA2U0VMRkJCbE1HeFJXaGpyYWplM3BiT1ZLditaNFg0MnE1d0NjMERINFZz?=
 =?utf-8?B?ZkRybDRHUFMwK3V3WVZIRDYzVkpyMEoxYUFNWDZuSVBBQlpPdEVnaUhZQkVy?=
 =?utf-8?B?NVB1Qm5LRHN1b1p6dnNzdVN0Y0tMWndrd3ZydXEzejFzemVBRzNIRnl0VFli?=
 =?utf-8?B?NmpUdXlUTU9BbjNuMjVoN3BmRUIvdUlSbzFoWXFjWFVVb1llK3RLOUQwRkZz?=
 =?utf-8?B?YjYwOTYxQlZQMkhzaS9rZHROOHVsYmJScUhDaUVyd1MydVlrZWFtajRhZVVX?=
 =?utf-8?B?bzV1cVdqVzZrODlVbWlXdDlhSlY5RXkrTndnajJaVGtZQUpoUEYrak9IZk83?=
 =?utf-8?B?Y210ZFkwdGdTRm53RVQxYmZ0NDQ0NFRRbElRL0Q4U3Z1Z2pGZURLMlp5c0xK?=
 =?utf-8?B?NlAybkQ0d3YxdXl0dytNdzdPd0RHdk1EZnhDdWFlR2NIaUtWTVRjSkF5U0pF?=
 =?utf-8?B?MmtjRmluQ0M1NldhZnFaM0FkL3kwZW5FM0QwRnZEZHo4dlZCdjFNOU9nMXJW?=
 =?utf-8?B?N3ZBcVUxMzdwVzVDZkg0YVJmUlBxckQ3cU5hTnJ6U0daY2lhTkxJeDA4cFhK?=
 =?utf-8?B?UmtxWkllSnRsYXBXK3R4bmlweGpRc2hyMGhnYmlmeVlCc2FIRHNzZENxc0hO?=
 =?utf-8?B?cTBJTGxrV05weTlMRXFKVWFIWTN3YnBkeEJuREl1bHlvWG15WHpsLy9URHRm?=
 =?utf-8?B?dnRqK1JKWXE4Q21ZM3p3dUQyUzcyVnU2a1RZVS9SS0JwSVBabmFvWHZFYnNW?=
 =?utf-8?B?ZS9RK2VpMU1aL3d6T285OEJWaGR2S1JnQmZDdUYwRlVGQUgvb0VwTk9mRW00?=
 =?utf-8?B?TE5PRnpSb3R1b3ExRjNFa3V5aFdnMEtabkFLVUFTbWtpZUVJYVV4WGl2bTQ0?=
 =?utf-8?B?cnlJREI4RmdwVXYxM1FHbk44M2RsSjVBY2ZNN1cvbkF2cDFNTzIyc1ZWVFdS?=
 =?utf-8?B?aEpVaGNjYk9zMXZtUFVFemh3S29SaTJWd0JRYlZCb2hZZ3JZSzlpOWhXeExu?=
 =?utf-8?B?OFRuL25EUkVXYXNRL1hiTzFMbnJXQm1BUnFaTE9yNllnMEpJR2dpOCtQS1lG?=
 =?utf-8?B?SDV3eUd2WmpWVmVwTlNjSEVMZzY4ZjZ2azZMUGQyajJHMm9Pd05JOG5PSE42?=
 =?utf-8?B?OGgrcUhVRHh3M2F2TkxmMithQXRnPT0=?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 83d9c98a-fa39-48ba-bdb8-08d9afb9c29d
X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB5009.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Nov 2021 02:17:42.0515
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: xybOlaEKm0/vN7QoyN6rHZk1OlVLsNPqBCANyikCDVgEQeKOJYmUtJSN6pYkmZGxT5CGZIVcFQvDSCvL4UuKecyLLQDALmK6LdLW6xDl/z0=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR10MB4333
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10178 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 spamscore=0
 bulkscore=0 suspectscore=0 mlxscore=0 mlxlogscore=999 phishscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000
 definitions=main-2111250010
X-Proofpoint-ORIG-GUID: OQhjtSONJBqWvYheHkHfOJjV7zgfYSpb
X-Proofpoint-GUID: OQhjtSONJBqWvYheHkHfOJjV7zgfYSpb
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 11/24/21 5:54 PM, Thomas Gleixner wrote:
> On Mon, Nov 22 2021 at 16:47, Sebastian Andrzej Siewior wrote:
>> From: "Longpeng(Mike)" <longpeng2@huawei.com>
>>
>> A CPU will not show up in virtualized environment which includes an
>> Enclave. The VM splits its resources into a primary VM and a Enclave
>> VM. While the Enclave is active, the hypervisor will ignore all requests
>> to bring up a CPU and this CPU will remain in CPU_UP_PREPARE state.
>> The kernel will wait up to ten seconds for CPU to show up
>> (do_boot_cpu()) and then rollback the hotplug state back to
>> CPUHP_OFFLINE leaving the CPU state in CPU_UP_PREPARE. The CPU state is
>> set back to CPUHP_TEARDOWN_CPU during the CPU_POST_DEAD stage.
>>
>> After the Enclave VM terminates, the primary VM can bring up the CPU
>> again.
>>
>> Allow to bring up the CPU if it is in the CPU_UP_PREPARE state.
>>
>> [bigeasy: Rewrite commit description.]
>>
>> Signed-off-by: Longpeng(Mike) <longpeng2@huawei.com>
>> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
>> Link: https://lore.kernel.org/r/20210901051143.2752-1-longpeng2@huawei.com
>> ---
>>
>> For XEN: this changes the behaviour as it allows to invoke
>> cpu_initialize_context() again should it have have earlier. I *think*
>> this is okay and would to bring up the CPU again should the memory
>> allocation in cpu_initialize_context() fail.
> Any comment from XEN folks?


If memory allocation in cpu_initialize_context() fails we will not be able to bring up the VCPU because xen_cpu_initialized_map bit at the top of that routine will already have been set. We will BUG in xen_pv_cpu_up() on second (presumably successful) attempt because nothing for that VCPU would be initialized. This can in principle be fixed by moving allocation to the top of the routine and freeing context if the bit in the bitmap is already set.


Having said that, allocation really should not fail: for PV guests we first bring max number of VCPUs up and then offline them down to however many need to run. I think if we fail allocation during boot we are going to have a really bad day anyway.


-boris