Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Message-ID: <71032a38-e1ab-ac73-09f7-9eefffd53674@intel.com>
Date:   Thu, 20 Jan 2022 08:28:36 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Firefox/91.0 Thunderbird/91.5.0
Subject: Re: [PATCH] x86/sgx: Silence softlockup detection when releasing
 large enclaves
Content-Language: en-US
To:     Jarkko Sakkinen <jarkko@kernel.org>, <dave.hansen@linux.intel.com>,
        <tglx@linutronix.de>, <bp@alien8.de>, <luto@kernel.org>,
        <mingo@redhat.com>, <linux-sgx@vger.kernel.org>, <x86@kernel.org>
CC:     <linux-kernel@vger.kernel.org>, <stable@vger.kernel.org>
References: <1aa037705e5aa209d8b7a075873c6b4190327436.1642530802.git.reinette.chatre@intel.com>
 <e4e8fbe757860cd24e2f66b25be60d76663935d8.camel@kernel.org>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <e4e8fbe757860cd24e2f66b25be60d76663935d8.camel@kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?U28yMmR2L0NoNGY1UmVNYnNHWTFGZ1p0R05XWHV3bEVISFJkMk44bFhGamRs?=
 =?utf-8?B?RWkrNG1wSXpScXE4V1Y2Y0VzMEtodUEwdTFrK1RIcUEwZkdqVTZRanMrY2tv?=
 =?utf-8?B?SXFvWVJTTFdiRVpvZ0FzODRpVU04NXRWeFpmOVFMWWxZbWI5QlF5WitOQjNB?=
 =?utf-8?B?QzBaVEd3aGhlMWlSUmpMSm5xVDBUdW15eXd3QXBqaDJrb2pMYkxlZUNOYlBz?=
 =?utf-8?B?TnRiMEo1WE5taWo0SjFBTTBUdE5BYkJTcVRjNytPZGNVN0h6UTFNTEY4cW50?=
 =?utf-8?B?UnVjZDFKZUNKd0gwTmJLV3QwcFRHZ0JOaW9tdG8xemo4WCtmcVVQc2hjNUdX?=
 =?utf-8?B?TEpldlhTVFllOTlTWVFzTkVVNGlKM0JiellyRXZSdFJ2eWxsMXRodjgvQXdv?=
 =?utf-8?B?Qmt4T2JzbTR2d1dtcmdZZytuMG5JQ3IwOFhuWmQycm1vcmt5TEZDWDBSWXNW?=
 =?utf-8?B?SHNQRm1WVEhEd1JKSzQ2ZFQwOTI3T2p3NUpOWGZXbGorNFgvS050d0pYUGsz?=
 =?utf-8?B?bEJLMENTNkFnSHdaeHVaUThmYVlpbDI4cTJxSE51TTVVQmtEQm5oVXd0SGFK?=
 =?utf-8?B?MWhJK2JZNk9yRmxzT0N3cmNNQmRvbm9ac2lxTVdXZ3RreC9tS0U2V2ROQlhL?=
 =?utf-8?B?S1N3eld2WjJuTWl4dXZpQWRQeGx3eUg2WHQ3cWdFREN4NHRtc0xFM3lnTmZT?=
 =?utf-8?B?UHRGUDJxZTREbDhFckUrMWJTWGMwWU96amtsV0kxVjhBNnJaQjhwRjV3aHNU?=
 =?utf-8?B?Y2IydmVWVTJ5d0RZRHU0dkxRNHRFdmh6MXpSb1ArYk1SYUQ4d0paNjhrVDd1?=
 =?utf-8?B?Y2pVaXNxM2Jlc0RCbEk4NXZDUlpTN3R6RVlCdjJ2MU01Zk4zY1drcmlPQXJU?=
 =?utf-8?B?T3g3cEZmbDRNa1JZelk4WkFmT0tSMVlDdE0vZGhTMUtLS1RYcjN6bFM3ZzNt?=
 =?utf-8?B?ckovNTc5VEZZZWFMUkt6VEVLT3g2aXRONEI5OXEwekVrUWpzWWhQOTUzQXZ3?=
 =?utf-8?B?cHNiRTRGbTJ4QjIyNkxuSUx5eTY5WU1YTmdLS0VRR1FSbmJZUUF0UFl2bWZG?=
 =?utf-8?B?RXBNQkovNG56L2lxdDFUTzRlN3NsUzR0N093dlJpekdNeFVLWkwvVmRWSFJl?=
 =?utf-8?B?WjNWejVkRmJ0S0Y5OTZRYmRnbWtLWGhwWGNiZ2VJN3dBc0FndXg4Umk1RmRB?=
 =?utf-8?B?WGo3eFVPN2Z5UXFHSkRRQzhiT0pTTXVVVUFZODMxcFk1UnUwQ25kRFYvUllj?=
 =?utf-8?B?cjNFTGRUM3UvUURkeU1IRWp5ejJpQWcyU0hING9zT1lDM0ZWUkQxendlNW0z?=
 =?utf-8?B?bWdvd3gwTUhFSis3S3ZqeEU0Q1F5aVAwYTdERW5YZkJ6eStWMkFsOU92ZSts?=
 =?utf-8?B?MmYxTEtKM2NMTTQ0Z0FsNGNianlHbXQ4Tm5SNlhvbGRucGo1MlhMMmdncEZT?=
 =?utf-8?B?cUtJdEJneElhZzRCWFNsVzdWcFduWUZWK21FbWVYckRjYlVoaUF4Z3YzeHpr?=
 =?utf-8?B?RHRNYmZ0RU13V1NqRk94S0xpUVRXVEJVeW9JU21nb3VzWnZlVGZLMy9ZNUls?=
 =?utf-8?B?c2JYcWkxb1dRQmIvSkNPQUNhaUpaM1VST0NhY2ZkMUVZL3p6RWJVOFZWU1pz?=
 =?utf-8?B?TVNSVGY3UTVYSVJuYXJ6YWlLYnh2UW5HUi9vb3g4L3JYUFNBbEtWM1ZBSHJq?=
 =?utf-8?B?b3hDNnBzZFROcld2S2NCVnNRZHJWeWlQSGdSMVlZTkNpNVBTK01YTFdjQmlK?=
 =?utf-8?B?K0h3UDJYeTErd1AzTmtYKzZrakFQdmNHNWo4VHdUK1Zwdm1ZcHc4TS83VkQ0?=
 =?utf-8?B?Snk1andOaVNwbTRQaVJ4WWZxMU9RWjZTQ3lpazZLdkZhY0tZT2xmbmFZRjlO?=
 =?utf-8?B?cXBYcXJPM2ROZzU5VTFyMm01SGx1MExnSGNvUnNmaEYzN1NGcXVUbDF0MUF2?=
 =?utf-8?B?ODNRb2Z5dyt1bit3NnhiN0VQZUMvRWZhYnpMN2J5ZTlQMkM1Ynp6Z3ZOYVlr?=
 =?utf-8?B?QmJMTGxnNUpJUHBPdk95M1VXOUtORWRINXc0V2NLQ01CNVY2RVd5QUFXVlQw?=
 =?utf-8?B?SXBkcjZyZVBmc0JLZlFCRFE3cm1aOG95bnZ4ZTNNOHN1U3pCVnVVY1Z4QXJO?=
 =?utf-8?B?cHBla002Zndua3p3YXdDV1FvcmNhcitEZjRvLzJSQVhaTzArei82K1cyYmRy?=
 =?utf-8?B?MkNhRjBwSTZZU3ZCOXpKSjBPcUxrS3BJS3VkdGpWQUpXVmhCdlRRL3pRdk9R?=
 =?utf-8?Q?+o834+boowQDrlGirP6cAeyHDQtCmAp5imXOBIVMSI=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: bab4e46e-5550-42cb-e534-08d9dc31eb31
X-MS-Exchange-CrossTenant-AuthSource: BN0PR11MB5744.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jan 2022 16:28:40.9871
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: gwhv+MJWQqOjmIEuIP/9aaiiDx9+WyIiGw++WPbKkBKT2KGWINHp4cB6qcgjkVF7pitc96vxkmEnIIahfU5vc4dewz772+RKeIUNcTfT+HM=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR11MB3911
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Jarkko,

On 1/20/2022 5:01 AM, Jarkko Sakkinen wrote:
> On Tue, 2022-01-18 at 11:14 -0800, Reinette Chatre wrote:
>> Vijay reported that the "unclobbered_vdso_oversubscribed" selftest
>> triggers the softlockup detector.
>>
>> Actual SGX systems have 128GB of enclave memory or more.  The
>> "unclobbered_vdso_oversubscribed" selftest creates one enclave which
>> consumes all of the enclave memory on the system. Tearing down such a
>> large enclave takes around a minute, most of it in the loop where
>> the EREMOVE instruction is applied to each individual 4k enclave
>> page.
>>
>> Spending one minute in a loop triggers the softlockup detector.
>>
>> Add a cond_resched() to give other tasks a chance to run and placate
>> the softlockup detector.
>>
>> Cc: stable@vger.kernel.org
>> Fixes: 1728ab54b4be ("x86/sgx: Add a page reclaimer")
>> Reported-by: Vijay Dhanraj <vijay.dhanraj@intel.com>
>> Acked-by: Dave Hansen <dave.hansen@linux.intel.com>
>> Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
>> ---
>> Softlockup message:
>> watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [test_sgx:11502]
>> Kernel panic - not syncing: softlockup: hung tasks
>> <snip>
>> sgx_encl_release+0x86/0x1c0
>> sgx_release+0x11c/0x130
>> __fput+0xb0/0x280
>> ____fput+0xe/0x10
>> task_work_run+0x6c/0xc0
>> exit_to_user_mode_prepare+0x1eb/0x1f0
>> syscall_exit_to_user_mode+0x1d/0x50
>> do_syscall_64+0x46/0xb0
>> entry_SYSCALL_64_after_hwframe+0x44/0xae
>>
>>  arch/x86/kernel/cpu/sgx/encl.c | 1 +
>>  1 file changed, 1 insertion(+)
>>
>> diff --git a/arch/x86/kernel/cpu/sgx/encl.c
>> b/arch/x86/kernel/cpu/sgx/encl.c
>> index 001808e3901c..ab2b79327a8a 100644
>> --- a/arch/x86/kernel/cpu/sgx/encl.c
>> +++ b/arch/x86/kernel/cpu/sgx/encl.c
>> @@ -410,6 +410,7 @@ void sgx_encl_release(struct kref *ref)
>>                 }
>>  
>>                 kfree(entry);
>> +               cond_resched();
>>         }
>>  
>>         xa_destroy(&encl->page_array);
> 
> I'd add a comment, e.g.
> 
> /* Invoke scheduler to prevent soft lockups. */

I could do that. I would like to point out though that there are already
six other usages of cond_resched() in the driver and it does indeed
seem to be the common pattern. When adding this comment to the now
seventh usage it would be the first comment documenting the usage of
cond_resched() in the driver.

> 
> Other than that makes sense.

Thank you very much for taking a look.

Reinette