Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <a4eb5ab0-bf83-17a4-8bc0-a90aaf438a8e@intel.com>
Date:   Tue, 15 Nov 2022 17:00:08 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.4.2
Subject: Re: [PATCH 01/26] x86/sgx: Call cond_resched() at the end of
 sgx_reclaim_pages()
Content-Language: en-US
To:     Jarkko Sakkinen <jarkko@kernel.org>,
        Kristen Carlson Accardi <kristen@linux.intel.com>
CC:     <dave.hansen@linux.kernel.org>, <tj@kernel.org>,
        <linux-kernel@vger.kernel.org>, <linux-sgx@vger.kernel.org>,
        <cgroups@vger.kernel.org>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        "Thomas Gleixner" <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>,
        "Borislav Petkov" <bp@alien8.de>, <x86@kernel.org>,
        "H. Peter Anvin" <hpa@zytor.com>, <zhiquan1.li@intel.com>,
        Sean Christopherson <seanjc@google.com>
References: <20221111183532.3676646-1-kristen@linux.intel.com>
 <20221111183532.3676646-2-kristen@linux.intel.com>
 <Y3QgZ+ZKAfJ92U5L@kernel.org>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <Y3QgZ+ZKAfJ92U5L@kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?OGUwYndra3lQM29RTWEzU1RFZm9LUGxDWWI0V282cE1ZTk1zMWRYL2lnT0Ew?=
 =?utf-8?B?MkVwRWhWQ0MyL1ovZnBoaWxKRi9FQTBHb3RRemdidmtXWno2TjNaMlhRbDV0?=
 =?utf-8?B?ZFFlNU9PMk1XSFFDTW9Xdzh4d2MzOHJSaitNOEVhN3J0WWtmL2tpdW8wTkRM?=
 =?utf-8?B?NmlkQytOcjdUQVNkc1U5YlVTT2E4K2h4NnZkb2xVdG0zbUFkK0ZOdHoyV2lr?=
 =?utf-8?B?NWJlMFJWUUU4K3VoYWRINWVGeStrVmhxeTQ4Mk44ZmZ6ZTVJbysyMlM4Qzd2?=
 =?utf-8?B?TXk4clpoVk5MVDVBdVNFMnNlUkRXUk1Fa0NrQmE4U1JVN3RDUEZnRm9yUzdx?=
 =?utf-8?B?aHpxUmlJcG1NSDJJVTJiZVdEQXQ4Vmc0Z2ovb2Q3SjFoVHMxZzBSK0sxZk1l?=
 =?utf-8?B?QUFDTDN6RHhTdmdIYy8zVFRMTFdiR3EwNFVxOHhDemlib2VjQnM3eDJVak9V?=
 =?utf-8?B?RlkzQWN4SlYyWjFIVjNidHJOY1AyNk02ZEh0VnFyVk5kYmloY1hrb1ZDVFFJ?=
 =?utf-8?B?anYzRFhzZ3JOUFRoQnpNOU1xWS9reHlzcGZES2t0N2oxRHNZeVcvL2xRTVY5?=
 =?utf-8?B?SmtHMUU4akM4a2ZtbHh6di9vUHpUYlprdXR3dUNKY2dIY3d5WURWcU5IL25K?=
 =?utf-8?B?b2V2NjdoYmZvV1VHVGtGckpYbVpycjFMUFRKdTF0UjZ3MjlQUzNHdDdKWjNx?=
 =?utf-8?B?YzRtS1dhNXgyYzRHN0ozVEdmenUxeVo1MW5PM0xNRjhJSlFLM1JmblptT21v?=
 =?utf-8?B?dlFhWHV0S1pMOVZ3dW5CRDFIUlY2dStDajNVQnBrYkhTSEpYeGhPZnNodUE5?=
 =?utf-8?B?b1N4UXlsa0UwalFYNEhzMEN4NHZvMHJWNGxaOExVK1dYcVVDaEFZZmNjT21Q?=
 =?utf-8?B?bnJUanlEY092Z2t1bldxc3RHb25mRjc4aHNCZU5mbkNOcEFjZTh0WVYrcmxU?=
 =?utf-8?B?NzdiR1k1VllsaWwrSm5CTlF6WVpxMDc3UDZHN0ZWYlN5dHR2UG5UZnQ1RjRJ?=
 =?utf-8?B?NENJRmpuYWJ0elpnNHhFU0JhM2dlNjVzUlRzL2pSMVFoYlBkM0dqblRxTnNy?=
 =?utf-8?B?bFdBYktiN0RqT2VSK3N5ZkYzUjJ1UHFiOUhlK29YVU1nMWFqU2lkK3Nrazdi?=
 =?utf-8?B?Sk5oOEg0VlFJWlVRQWhiOTFNYi9rNnFDZEdlR0ZoVytPS3BLSmJpZ1E3N1RO?=
 =?utf-8?B?TjVDcm01bjZuRFNYK21HbERjZlArOW16SlF2dUNlK3laRzRZLzd5Z1ZwVVdl?=
 =?utf-8?B?alE5MS94OG5IN0hYejdzVTJqSU1MQ3ltT0dLWVJpWkRXMGpEQWc3ZnBlT2tL?=
 =?utf-8?B?MXI0UjhndTFma3d5dklWTjljTW5jaGRBNE9hMlFNSVIrb0ZMUEx4YXl3Z2t6?=
 =?utf-8?B?SGo2RXRkd25MOUVCRU1YVC9yNUlZZitTVUlSQkRKS0NwTWRIbUJ4ZDJFblp0?=
 =?utf-8?B?QXdXU2k4eHJNM2d0dUhzK29ycjVCMTE1U082MUJkZ1g0d2VITWN3bTlpanR4?=
 =?utf-8?B?ckZZTFA0TDZKVCtXWW5zOVU0ekcrakwyTlJ1NUFnVDJrUXJyOGVReDdpZlZK?=
 =?utf-8?B?SXVTdExjSlRHQ3lsZGlYcUZ0akxUeHoxZGRzSTdLMGdHU29sc0tpQmx4a1Vn?=
 =?utf-8?B?RGNkSEJuTFBMb3ZJQ1Fkb3Z3NldhUnV3NXg5QXZ5VzkxWnRmbGxHVHRYSzRU?=
 =?utf-8?B?aFFUb1NoY1BCSXQzMEpNbTludlhiSUs5UU5QblU3TkxKU3UvYlNMampGSG42?=
 =?utf-8?B?QjEvY1FUcUtKWnFOVzVleFcxN05KTmpIc1dJRE9tczdqdWk3ZDZpK001S053?=
 =?utf-8?B?WVlCcm4zY1BoejJybDNveGk1SURDQ0U4ZFlNenNwbmhoL1dzeCtUWmZLWHA3?=
 =?utf-8?B?U2h0MkhrRHphWk9YK29YWkVIMGFuNjBEZ3huT3VOWUFvTmRMeXZQaWRUNGhM?=
 =?utf-8?B?U0c3L0N4REVZd2pKdTJjc3Nvdk9KVzFXSzhqL0J5Y3JuV1cvY3JZYm9Ka3A0?=
 =?utf-8?B?UHk2WkV3amloM1ZyUUxoZDFTZEVXcjVaekVKMjVtWCtRTGR0eEorVmRFK3FV?=
 =?utf-8?B?ZDNTZDhtMHp4UkFpTURVeUlUdHZIMjBzYlk3aUlrb1pqY21GSXZwY1VTUmJr?=
 =?utf-8?B?UUFFUTcxZkY1NjhDbkNwQk5RR25nUllPTUFzamY1REFwallhVi9nVkFGTGhY?=
 =?utf-8?B?S0E9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 7d674a80-d4e3-4825-410c-08dac76de9dc
X-MS-Exchange-CrossTenant-AuthSource: CY4PR11MB1862.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Nov 2022 01:00:12.8828
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 0u09Jd0zvDryQLDEl/IDn9CjQF9ESljZYnlkssMN9L3IfEPzG11qto68uQbXaJSNVTr/UUnCd+imqbO/MXACSHvBAtDBsZXP4VuPA1A+56w=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA2PR11MB4892
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_HI,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Jarkko,

On 11/15/2022 3:27 PM, Jarkko Sakkinen wrote:
> On Fri, Nov 11, 2022 at 10:35:06AM -0800, Kristen Carlson Accardi wrote:
>> From: Sean Christopherson <sean.j.christopherson@intel.com>
>>
>> In order to avoid repetition of cond_resched() in ksgxd() and
>> sgx_alloc_epc_page(), move the invocation of post-reclaim cond_resched()
>> inside sgx_reclaim_pages(). Except in the case of sgx_reclaim_direct(),
>> sgx_reclaim_pages() is always called in a loop and is always followed
>> by a call to cond_resched().  This will hold true for the EPC cgroup
>> as well, which adds even more calls to sgx_reclaim_pages() and thus
>> cond_resched(). Calls to sgx_reclaim_direct() may be performance
>> sensitive. Allow sgx_reclaim_direct() to avoid the cond_resched()
>> call by moving the original sgx_reclaim_pages() call to
>> __sgx_reclaim_pages() and then have sgx_reclaim_pages() become a
>> wrapper around that call with a cond_resched().
>>
>> Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
>> Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
>> Cc: Sean Christopherson <seanjc@google.com>
>> ---
>>  arch/x86/kernel/cpu/sgx/main.c | 17 +++++++++++------
>>  1 file changed, 11 insertions(+), 6 deletions(-)
>>
>> diff --git a/arch/x86/kernel/cpu/sgx/main.c b/arch/x86/kernel/cpu/sgx/main.c
>> index 160c8dbee0ab..ffce6fc70a1f 100644
>> --- a/arch/x86/kernel/cpu/sgx/main.c
>> +++ b/arch/x86/kernel/cpu/sgx/main.c
>> @@ -287,7 +287,7 @@ static void sgx_reclaimer_write(struct sgx_epc_page *epc_page,
>>   * problematic as it would increase the lock contention too much, which would
>>   * halt forward progress.
>>   */
>> -static void sgx_reclaim_pages(void)
>> +static void __sgx_reclaim_pages(void)
>>  {
>>  	struct sgx_epc_page *chunk[SGX_NR_TO_SCAN];
>>  	struct sgx_backing backing[SGX_NR_TO_SCAN];
>> @@ -369,6 +369,12 @@ static void sgx_reclaim_pages(void)
>>  	}
>>  }
>>  
>> +static void sgx_reclaim_pages(void)
>> +{
>> +	__sgx_reclaim_pages();
>> +	cond_resched();
>> +}
>> +
>>  static bool sgx_should_reclaim(unsigned long watermark)
>>  {
>>  	return atomic_long_read(&sgx_nr_free_pages) < watermark &&
>> @@ -378,12 +384,14 @@ static bool sgx_should_reclaim(unsigned long watermark)
>>  /*
>>   * sgx_reclaim_direct() should be called (without enclave's mutex held)
>>   * in locations where SGX memory resources might be low and might be
>> - * needed in order to make forward progress.
>> + * needed in order to make forward progress. This call to
>> + * __sgx_reclaim_pages() avoids the cond_resched() in sgx_reclaim_pages()
>> + * to improve performance.
>>   */
>>  void sgx_reclaim_direct(void)
>>  {
>>  	if (sgx_should_reclaim(SGX_NR_LOW_PAGES))
>> -		sgx_reclaim_pages();
>> +		__sgx_reclaim_pages();
> 
> Is it a big deal to have "extra" cond_resched?
> 

sgx_reclaim_direct() is used to ensure that there is enough
SGX memory available to make forward progress within a loop that
may span a large range of pages. sgx_reclaim_direct()
ensures that there is enough memory right before it depends on
that available memory. I think that giving other tasks an opportunity
to run in the middle is risky since these other tasks may end
up consuming the SGX memory that was just freed up and thus increase
likelihood of the operation to fail with user getting an EAGAIN error.
Additionally, in a constrained environment where sgx_reclaim_direct()
is needed to reclaim pages an additional cond_resched() could cause
user visible slow down when operating on a large memory range. 

Reinette