Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <26d0e4cc-be0e-2c12-6174-dfbb1edb1ed6@oracle.com>
Date:   Wed, 20 Jul 2022 12:38:50 +1000
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.11.0
Subject: Query regarding deadlock involving cgroup_threadgroup_rwsem and
 cpu_hotplug_lock
Content-Language: en-US
References: <8245b710-8acb-d8e6-7045-99a5f71dad4e@oracle.com>
From:   Imran Khan <imran.f.khan@oracle.com>
To:     "tj@kernel.org >> Tejun Heo" <tj@kernel.org>,
        lizefan.x@bytedance.com,
        "hannes@cmpxchg.org >> Johannes Weiner" <hannes@cmpxchg.org>,
        "tglx@linutronix.de >> Thomas Gleixner" <tglx@linutronix.de>,
        steven.price@arm.com,
        "peterz@infradead.org >> peterz"@infradead.org
Cc:     "cgroups@vger.kernel.org >> cgroups"@vger.kernel.org,
        linux-kernel@vger.kernel.org
In-Reply-To: <8245b710-8acb-d8e6-7045-99a5f71dad4e@oracle.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?a0tFS25pbTYvY1ZVU1ZiRFUvWDRPL1JBU1V6UUFhT0hZMUxqRll5WXdOT1Ry?=
 =?utf-8?B?cU9ZQzBzeEF0cTNsa1plR2lCSENVbEVQbEFGNGFEdlBFdE1COFR2VFJMWWxq?=
 =?utf-8?B?d3VOK3JHeVhDeE9sbnFmUnFCUFdvbFZicC82MWViMFFJb29wZ3dSTnVSYlV6?=
 =?utf-8?B?YVNycnIxdWpRNERINEVramFKa2lVL2pWd28zNkpSbnYzN1VJWGFZa0FtcVRC?=
 =?utf-8?B?R1liL1I4aGk1RTN1WkxncndXcy9JNmlSWllhSSsrWEpKNjBTUmdxenlrbzhM?=
 =?utf-8?B?bVdyTXQ1Yk5pajh0Q3BpQzRBR3QxNEhtaUNPVTlyeFRXYXFxWEZxTVUyZnNp?=
 =?utf-8?B?OFcvRkVmdDIrRnJhRWxTSXM0NVlpbGNxbDRvbDZFTS9TcHM5ZDhocHluc0ZL?=
 =?utf-8?B?U1hHWXVuLzhRNnFGZ2hFRy96cWx4dENpazRiVWtyNEVLbGpQUTlpSy9aeExv?=
 =?utf-8?B?c09RVXZhUHUybmhpK2w0NXozOXo4WXNWNE1sUUIzT1N0L2g0a05mb1dOQXhv?=
 =?utf-8?B?Um9LbzVzRzhhcnhWVGR6NTB4ZDRwREE5OTZPaXk1VmpDYzU3ZVJBRGFkR3VM?=
 =?utf-8?B?bzVTbkZ1S2JyOW9xdjdhSGFsZFhRYk1CZkdnS29nVEgrTEJMVzFOVkxROHJP?=
 =?utf-8?B?NlRxN3NONnlYTWw3UU1GeE9mSHhBbG5ZYmJLTno5SHdSb01ja2VHMmJMbjhY?=
 =?utf-8?B?L1NJcUFGS01FNm9GMm1ZL3I1OFJrK3Rnbjh0dE92NzJqYkxtM3dVL1RKZFpJ?=
 =?utf-8?B?Ym1IS1R3bzlVVUR6T3I4TWhOVEgwN3VkTFd0cEJOaXZ4TEs5bWR1elY0d0gz?=
 =?utf-8?B?VUM0TDVZSE9oT3lnc0JSYU1UV0YzOWdjZWhadWgrd1dpNzhwYTZxdlZkTHJt?=
 =?utf-8?B?Q2lKUnVpU3g5VllnZUNFQktpNUxreFBzc0pqbWR1dlNJZ3BTRUFKQU5aWDdp?=
 =?utf-8?B?U2hOWkhFbGpmU2VZNVJGbzFJekpkcmdXMHZMSWFWc2VqcDh5R3hHWGRIMHM3?=
 =?utf-8?B?QlRkK3g5bDRWTnVsak85RWxOKzhadWl3RXMrcjdoeHpKRzNsWmY2Wnk5MWsy?=
 =?utf-8?B?TThVT0NmeUdXWXAwWGlITUNHUHlaK1VWamE5UWdaaG5CQlJrTXc2U2dadklJ?=
 =?utf-8?B?d2lxUDY0dW1ySkY3RG1TMEFjbWE2ZDJXRHpnc0VNbWtYY3lFRFgrQVdVQ2d1?=
 =?utf-8?B?eFhZaTEvRGdNZjJFR0dxUzdMSE9pTnR0VlkzeUdCYVkxUUxTb1dQYUhYbHNq?=
 =?utf-8?B?QzJyQXB1VGI0dENDYTZvWlVKM0tqOGhzUUZDL1h3UnRUQXhzVmFaRFBiMlRh?=
 =?utf-8?B?K2Y3VnJOUjNGdVZyWkR4eDUxR0d2Qy9jY242Y2Jia1hBR2pUU2JkWStSZVVo?=
 =?utf-8?B?azBnRGtZbERkcEFVN1pMTVNqREwvd09Jc3pERzV2QTgrejV1bFNvZXY1Si94?=
 =?utf-8?B?Zjl1SGZuNUk5b0VCTUJ4UUlENktsOFRQN2FDQm1POUYwcFlvTnBlM2lGN1M3?=
 =?utf-8?B?Q0NMblBsR0lETm1iTi8zZ2YrWWtpMXVkNCsrL3lROGRJUGQ5R25KSFF5VFpm?=
 =?utf-8?B?MG1VMjlSV1p0WkhlN3hYNUFQVElmS0lRTVpNVElqVjZVdEYwZ1FqVzBHejNH?=
 =?utf-8?B?Rmg2WUhlYk1hMUZjTGxGVWxRK3p0MVVrOEFtSnd5YWdLRDZ2eEdxL2x3SDEz?=
 =?utf-8?B?c3JvQ2hVcDFJeVZPYWIwR0d1TUt0MU1MYmpKSU82L2QrMDhZUkRiS1J5Uy9I?=
 =?utf-8?B?RVYwSlptQ2YzYUp5VkVkMi9LcTJmeTI5bXc2akg5NENPclJDRG5hVnd2UjVm?=
 =?utf-8?B?cUVTbzFZSUVDQnV5d2hDTUdTQUpkVWk3TnZhbTYwekx6ZjYzWko1MXBYVmR0?=
 =?utf-8?B?bXBwNDRXZkEvK2dhSE41RENFV0tvb1Jzc0NURDJ3NFdELzVGWHFxMFhYK282?=
 =?utf-8?B?cm5ybjJ4Q3QwR1JRQWVZL01zRVY4NzJNbzg2WEFZdnlvd2pDYVhBVjBTc2Zm?=
 =?utf-8?B?ZkhqTzAxa0E0Tk9XV3RQK2ZleXZuTHZjanhRbTNJd09HT0Mya3ZZM3J2OG1n?=
 =?utf-8?B?RFh2d3Q0THpJRDcvakJXd1NraWxyZUZWL2lkWlhMcXkyZmpTNmd1eTdxUWQz?=
 =?utf-8?Q?zQgN47GYK4BbnO2a3qFEauCzt?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: dd14c4ec-11e7-4771-1b6b-08da69f9018d
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4479.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jul 2022 02:39:02.2331
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: gIkOEXK2bEWqRiphrbbpyK/ii9u0HvCVGmY+4FckOUdr9j15esRMjE9mHXSlprpwpJZTd/eicHmst/q/jECF+g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR10MB1350
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1
 definitions=2022-07-19_10,2022-07-19_01,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 bulkscore=0 malwarescore=0
 mlxscore=0 phishscore=0 mlxlogscore=999 suspectscore=0 adultscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2206140000
 definitions=main-2207200009
X-Proofpoint-GUID: YH8z01YfBPYmlexsq20nB73j66s1zUsL
X-Proofpoint-ORIG-GUID: YH8z01YfBPYmlexsq20nB73j66s1zUsL
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hello everyone,

I am seeing a deadlock between cgroup_threadgroup_rwsem and cpu_hotplug_lock in
5.4 kernel.

Due to some missing drivers I don't have this test setup for latest upstream
kernel but looking at the code the issue seems to be present in the latest
kernel as well. If needed I can provide stack traces and other relevant info
from the vmcore that I have got from 5.4 setup.

The description of the problem is as follows (I am using 5.19-rc7 as reference
below):

__cgroup_procs_write acquires cgroup_threadgroup_rwsem via
cgroup_procs_write_start and then invokes cgroup_attach_task. Now
cgroup_attach_task can invoke following call chain:

cgroup_attach_task --> cgroup_migrate --> cgroup_migrate_execute --> cpuset_attach

Here cpuset_attach tries to take cpu_hotplug_lock.

But by this time if some other context

1. is already in the middle of cpu hotplug and has acquired cpu_hotplug_lock in
_cpu_up but
2. has not yet reached CPUHP_ONLINE state and
3. one of the intermediate hotplug states (in my case CPUHP_AP_ONLINE_DYN ) has
a callback which involves creation of a thread (or invocation of copy_process
via some other path) the invoked copy_process will get blocked on
cgroup_threadgroup_rwsem in following call chain:

   copy_process --> cgroup_can_fork --> cgroup_css_set_fork -->
cgroup_threadgroup_change_begin


I am looking for suggestions to fix this deadlock.

Or if I am missing something in the above analysis and the above mention
scenario can't happen in latest upstream kernel, then please let me know as that
would help me in back porting relevant changes to 5.4 kernel because the issue
definitely exists in 5.4 kernel.

Thanks,
-- Imran