Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
Message-ID: <e118f4c6-7216-15f3-2bda-3a5851b0bcb1@intel.com>
Date:   Tue, 5 Apr 2022 10:13:35 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Firefox/91.0 Thunderbird/91.7.0
Subject: Re: [PATCH V3 19/30] x86/sgx: Free up EPC pages directly to support
 large page ranges
Content-Language: en-US
To:     Jarkko Sakkinen <jarkko@kernel.org>
CC:     <dave.hansen@linux.intel.com>, <tglx@linutronix.de>,
        <bp@alien8.de>, <luto@kernel.org>, <mingo@redhat.com>,
        <linux-sgx@vger.kernel.org>, <x86@kernel.org>, <seanjc@google.com>,
        <kai.huang@intel.com>, <cathy.zhang@intel.com>,
        <cedric.xing@intel.com>, <haitao.huang@intel.com>,
        <mark.shanahan@intel.com>, <hpa@zytor.com>,
        <linux-kernel@vger.kernel.org>
References: <cover.1648847675.git.reinette.chatre@intel.com>
 <b3a17e1ce7bcd14db3c28903e8a97f094998ae74.1648847675.git.reinette.chatre@intel.com>
 <Ykvrk4hYvBEnNOOl@kernel.org>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <Ykvrk4hYvBEnNOOl@kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?VEpjblNoYU9Ta1NEcVZvQW5heFZqQUtncE5HZmNFV0FWWkEyUFdLOU04NnJa?=
 =?utf-8?B?MFJGek1scGhEV1gzNWMvUkFWYi9uMHFpSmJaaW0wbk1RWlcxQk9LbnNqaUd5?=
 =?utf-8?B?cTJFWW1pd21Tb3RwUWc1ZU92VkhlQmp3dERzK0lvUXpERnl0a0pkZFRPR0lW?=
 =?utf-8?B?UGNoQmx5ZHE3WDN6NGI1UnA5MnB4L3Jmc2FjUnMwUlhTanRMTXdzOVIrTHRW?=
 =?utf-8?B?SVBuekppNXNXV1hWU0haZjlLdVBqTkQxd3lBWUxrVlAxZkJUTHArMTJSZzVi?=
 =?utf-8?B?S1hHZXNOQkZ0REtpR3ZHZWdvR3lQRVR6d0lnaVdsQUZVMkFGbG1US01YTTJV?=
 =?utf-8?B?RzFWSXVSMDg3MWU1SGpUSlZ1UEl0YkFVVUZOQWEvSDJUWjRoTHNKb3F3SW5F?=
 =?utf-8?B?WkM3VHBnNXo2c3U5Zkh3YlJMbWVpNkdrd2JuV251WitTYWhlT3BoTG5tQmZW?=
 =?utf-8?B?OG54TWMzMG11Wk5lOFR2bVAzRGJGYTRNWXlHT2JhV2lpT0g5K2RNTGRkWlc3?=
 =?utf-8?B?UW4xMndvRlgxREU4ZmdmalhmeVdVeDdpc3RRbUlIVXlIY29Oejh5UFZDanc0?=
 =?utf-8?B?S2pocGlxRlFwcTVFWDFHU2I4bG82bDA0cmpWakRlYUluTDNBZVhhZndHZzUx?=
 =?utf-8?B?WWxQMTZXMnEyOWtqWVV0ZEFDNmNBNld0dlFaQ1dPN0tXK2NKaWVwT2tnREtK?=
 =?utf-8?B?cWpsdTNWTjNMUGZYV2pNL29ZOUpFbVdQZUFZbG9FSnpuQ0Ftam1mN2M3THA4?=
 =?utf-8?B?Q3d6TWxRczJjYkNEazU5My94a1dmaU1IQ2hMVVd4ektWbnZjUXU4dkcrbkRr?=
 =?utf-8?B?WEIvU1dTdW5oYndqRnNRa2ZUUzMvSUJLV0pLSzNXNDBYTWo3VlpmQ2dEMlU1?=
 =?utf-8?B?Ti9pN00wMEl6UnVndmVZdGF2d0pFSERDVlN5SEZQeDRySkFXN1djZHZMOVpC?=
 =?utf-8?B?Q1BtNklyN25IOFpWdWxJL2tKNENocjNSNkhBdTR2aDgxRmNwMG15MUN3Q3p5?=
 =?utf-8?B?UnQvOHFFVXVKemRIMzhrM2llYlJBdDNMbE9lUDVTSkQ0MEZNNC9lSVlzVytT?=
 =?utf-8?B?YzMxbENxTGNTMVpKdXBWYktpM0xwSjlQanA4TDdSck5TdkRyTUtyUnErN2tk?=
 =?utf-8?B?WGJJUGJudzF2WFNKM1VTNVF0cDhTQ2U1eFNpREFZeEtqbElLM295NUJNa2ZS?=
 =?utf-8?B?MW5uMHpzVjNDd1FZalNVQWtEYk9tNUYvTG9hNVV1MUZacVo1Ums0SjJDdkIr?=
 =?utf-8?B?ckx6eEFySjEwNVBKNHFmd0lJMFpYQnlFRGVVVldpcnd4UkVocWNrbXFaZFlp?=
 =?utf-8?B?dmNnNVBjRnp5ODRGVlBDL2lsL3FwS1YxVjFWdzlXTGFyZU1FZ3JOZ0lQNnZD?=
 =?utf-8?B?aHRhTkFzcTZUaXFCVXhHZWE4SUsxL0U5ZFpMSEpJUnJYZnQrc1RUNktKak8v?=
 =?utf-8?B?Z1F0K2QvS2p1Wm0vVkpJVEpldlNLZmJ0Z2pIY0Y1cWxyL0lnd2kvMks1YnA2?=
 =?utf-8?B?RGJ3d2RLOW53emRJckpRaVI3YkJ1aHJicExvc0tnTWJIRm1ubUk0SGtrRFhq?=
 =?utf-8?B?RDNFcldReDAwODFXU3ZjeEpLcVNNQW5PRHBGd2ZKZHA0MEI1M1JWdkEzNzh6?=
 =?utf-8?B?d2F6Y3N5dHJKaGRyb3NTWTVrK3ExQUtxUGRML3NvR3h2S3ZLcmJLd0ZiSWpL?=
 =?utf-8?B?MWlrL0NmRlRQMnBXTjZKeVZyZXlIWkI4UUpabHJ4ODVKUTUzekd6SDZDaXpr?=
 =?utf-8?B?S2dhbjk3RWNmVHplWmt0VE5zYnlWRkdSTTB5aVZYU0syYWdjODRzQUx1ZXVE?=
 =?utf-8?B?aWpDUWhTbVQxbDlGN1FVTlVRWWpBQThkbWxpU3ZaUFZRMlZmeXFLU2lkUThR?=
 =?utf-8?B?YVQ2VXlLa2RlYUh0MzBCSmpabjJRM2NxK2dJNHZGMWVCWkRGc0hZczFZVG1X?=
 =?utf-8?B?MmZUT01iR0lwTys4akI4Z1g3QXpHb0lLUHFmQTR0ZjNQbk9ZMDR3RXpXZ1c4?=
 =?utf-8?B?a3Q2dXNRVVFMeTlxVHZVK3BrNDRHWHVvRGQ4U1F1eWVnc2hFVGNEWC9RdjNw?=
 =?utf-8?B?RmV0ZWZjWUQzelR0R2FqU09zU1V3NXlZK245VmtqNkJHWFNGSnhCamNpVm9U?=
 =?utf-8?B?Vmd1QUZjamJFZHZGM0hpVHNTSFM3UnlienJvblVnRWp6V3g5c3VicTFaY3JE?=
 =?utf-8?B?dzJJOU1pSkVBMFhPTTZqMzV6QVZGQzBSSmZNQWxzemkwRkk2T3A1YTYrMjV5?=
 =?utf-8?B?aGNLSjdJNzFiSWJTU3hhUnp0TUY1Q05hZnR6VlA1aUJiRFBMOXI4SEpac05C?=
 =?utf-8?B?cmhjdU1sWHZOS3lSWm4wbGU2UTJ0UUVGVTc1dGdlS3JHcmtxUGw5RHJhUUcy?=
 =?utf-8?Q?oHrwFUt1xOSF7gOY=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 7a1ded85-a87f-46ea-c0be-08da1727a10c
X-MS-Exchange-CrossTenant-AuthSource: BN0PR11MB5744.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Apr 2022 17:13:40.2445
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: XSyP1QXEtoDvfU4vOQeQGioHb4JMWUYZ5u0ob79ycHQNEpkR9WZqDkEf03QwXt6vSD3wOy0ioEHx99lYsxPfZOHnzXRggeUCTC2NVCeSDmY=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR1101MB2289
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,RDNS_NONE,SPF_HELO_NONE,
	T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Jarkko,

On 4/5/2022 12:11 AM, Jarkko Sakkinen wrote:
> On Mon, Apr 04, 2022 at 09:49:27AM -0700, Reinette Chatre wrote:
>> The page reclaimer ensures availability of EPC pages across all
>> enclaves. In support of this it runs independently from the
>> individual enclaves in order to take locks from the different
>> enclaves as it writes pages to swap.
>>
>> When needing to load a page from swap an EPC page needs to be
>> available for its contents to be loaded into. Loading an existing
>> enclave page from swap does not reclaim EPC pages directly if
>> none are available, instead the reclaimer is woken when the
>> available EPC pages are found to be below a watermark.
>>
>> When iterating over a large number of pages in an oversubscribed
>> environment there is a race between the reclaimer woken up and
>> EPC pages reclaimed fast enough for the page operations to proceed.
>>
>> Ensure there are EPC pages available before attempting to load
>> a page that may potentially be pulled from swap into an available
>> EPC page.
>>
>> Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
>> ---
>> No changes since V2
>>
>> Changes since v1:
>> - Reword commit message.
>>
>>  arch/x86/kernel/cpu/sgx/ioctl.c | 6 ++++++
>>  arch/x86/kernel/cpu/sgx/main.c  | 6 ++++++
>>  arch/x86/kernel/cpu/sgx/sgx.h   | 1 +
>>  3 files changed, 13 insertions(+)
>>
>> diff --git a/arch/x86/kernel/cpu/sgx/ioctl.c b/arch/x86/kernel/cpu/sgx/ioctl.c
>> index 515e1961cc02..f88bc1236276 100644
>> --- a/arch/x86/kernel/cpu/sgx/ioctl.c
>> +++ b/arch/x86/kernel/cpu/sgx/ioctl.c
>> @@ -777,6 +777,8 @@ sgx_enclave_restrict_permissions(struct sgx_encl *encl,
>>  	for (c = 0 ; c < modp->length; c += PAGE_SIZE) {
>>  		addr = encl->base + modp->offset + c;
>>  
>> +		sgx_direct_reclaim();
>> +
>>  		mutex_lock(&encl->lock);
>>  
>>  		entry = sgx_encl_load_page(encl, addr);
>> @@ -934,6 +936,8 @@ static long sgx_enclave_modify_type(struct sgx_encl *encl,
>>  	for (c = 0 ; c < modt->length; c += PAGE_SIZE) {
>>  		addr = encl->base + modt->offset + c;
>>  
>> +		sgx_direct_reclaim();
>> +
>>  		mutex_lock(&encl->lock);
>>  
>>  		entry = sgx_encl_load_page(encl, addr);
>> @@ -1129,6 +1133,8 @@ static long sgx_encl_remove_pages(struct sgx_encl *encl,
>>  	for (c = 0 ; c < params->length; c += PAGE_SIZE) {
>>  		addr = encl->base + params->offset + c;
>>  
>> +		sgx_direct_reclaim();
>> +
>>  		mutex_lock(&encl->lock);
>>  
>>  		entry = sgx_encl_load_page(encl, addr);
>> diff --git a/arch/x86/kernel/cpu/sgx/main.c b/arch/x86/kernel/cpu/sgx/main.c
>> index 6e2cb7564080..545da16bb3ea 100644
>> --- a/arch/x86/kernel/cpu/sgx/main.c
>> +++ b/arch/x86/kernel/cpu/sgx/main.c
>> @@ -370,6 +370,12 @@ static bool sgx_should_reclaim(unsigned long watermark)
>>  	       !list_empty(&sgx_active_page_list);
>>  }
>>  
>> +void sgx_direct_reclaim(void)
>> +{
>> +	if (sgx_should_reclaim(SGX_NR_LOW_PAGES))
>> +		sgx_reclaim_pages();
>> +}
> 
> Please, instead open code this to both locations - not enough redundancy
> to be worth of new function. Causes only unnecessary cross-referencing
> when maintaining. Otherwise, I agree with the idea.
> 

hmmm, that means the heart of the reclaimer (sgx_reclaim_pages()) would be
made available for direct use from everywhere in the driver. I will look into this.

Reinette