Message-ID: <0ccd75d1-1595-f213-c8b7-95988ab1c842@intel.com>
Date:   Mon, 6 Dec 2021 14:07:17 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Firefox/91.0 Thunderbird/91.3.2
Subject: Re: [PATCH 24/25] x86/sgx: Free up EPC pages directly to support
 large page ranges
Content-Language: en-US
To:     Jarkko Sakkinen <jarkko@kernel.org>
CC:     <dave.hansen@linux.intel.com>, <tglx@linutronix.de>,
        <bp@alien8.de>, <luto@kernel.org>, <mingo@redhat.com>,
        <linux-sgx@vger.kernel.org>, <x86@kernel.org>, <seanjc@google.com>,
        <kai.huang@intel.com>, <cathy.zhang@intel.com>,
        <cedric.xing@intel.com>, <haitao.huang@intel.com>,
        <mark.shanahan@intel.com>, <hpa@zytor.com>,
        <linux-kernel@vger.kernel.org>
References: <cover.1638381245.git.reinette.chatre@intel.com>
 <4d3cc24f06ae75d99449f1d505cbff9ba53819a9.1638381245.git.reinette.chatre@intel.com>
 <Yav99E+wkPaQzH7N@iki.fi>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <Yav99E+wkPaQzH7N@iki.fi>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?TmtTMDFjV0ZhOGlHdmkyUjEzSHZsQ0dvci96QTJpVFRIT3RTb1dwUmdtWEZ0?=
 =?utf-8?B?ekhqYUIyUHN1bjRKTHhjK1FGZld0OVAwMnJrUlZpRlRIV2UwSW8rV214MkFq?=
 =?utf-8?B?QlFoVkdzNVF3UWRsU1hXMEVNVmRaWlp2RHQ5VHU4R0xGMDArY05tVGNST1pG?=
 =?utf-8?B?T1hjWE9yMWpISDZSOEJCRUp6cG01VU9ScFVPOW5ZWjZ0dGRaR28xSmdpY2Qw?=
 =?utf-8?B?TXRkditSUCtLdnV1Ym1zYzRRNFRrUFpDcnZxWXdiRUdnTncxTWVGTndNbDQ3?=
 =?utf-8?B?VEwxa2h5L2Q0NDIvMVMvUVlva3pPcEtWTjV0dlpYU3p4M0loejNHVnlZZTVT?=
 =?utf-8?B?cEJvVjlqOEJVTDRlNDllQnE4ZjNqZkZhb1BlV1dLeSt6RW94N0V1V0FNcURM?=
 =?utf-8?B?VmxTWXFuRFJTNzhNMUVHa21zWXhteUtmMnM2M29Hei93c3JxSStERmJPVkJx?=
 =?utf-8?B?Y25SOVNQNFBRcDk3cFNPdUVneXVFZGlXbG9mU3hMMld5aEM0YzRpaS9sbGxs?=
 =?utf-8?B?RDJTbVZqUytoa0pST1RFb3o2Um11MEI1d3NGeHhwRjBmaVo4RlQwb0kzSHpQ?=
 =?utf-8?B?UStOc1YwOU9TMklVdHlycE14cStzMFUyR2ZlcndhUDBPSExvM3NVR2RjTFdS?=
 =?utf-8?B?eUpDQlVjejQ4WUFiZVhxalh5eURKSytVbUFhcS9vYm5vYVdGMys3Ly82eWNB?=
 =?utf-8?B?U0NsWmRsRUFQRWE2LzY2YVNXZ2xLUGJORWVVdzArL0hkYTd1ME4xWG9pM0RY?=
 =?utf-8?B?cElCQkxONWp6V0Fqa21Ocm55cVlCT0NaQWtMbkplRUVZNmROZ2RZZjc2RC9h?=
 =?utf-8?B?aExiYjV4ekptQjBUOE5UeW5MOVhiYlpKeUpGVjRPQitGU0UyVEZSRFNEckc4?=
 =?utf-8?B?NWxaWVpIOVplMDBsRTFJQitmRmZCaUJVSTV1REdJZHczT1ZtTGN5N3MyVkwy?=
 =?utf-8?B?VmRpelcyc1JVM3AwQ0NEbjN5QjhDYndPU3lJSjErZHhxck9WNUNLSzd4a3ll?=
 =?utf-8?B?WERoc1JBL041WUdjRGgrb3BWL0V5TmJKZnBuMzhvaEVwVGJPTFdsYlZkTHBq?=
 =?utf-8?B?SEtCMWc2OGl6YXZBN1lCMFNRUDFLOGlEQ2U5NTNYcSszWUIzMkJMb2NobzMy?=
 =?utf-8?B?K0U1TFh3cU5rUXZJd3YxTFdETkE2NlNyN3ArajYxKzVrcCtvMHR5TlhoNXNm?=
 =?utf-8?B?Z3dhVWhzbG1rZjVPNEpwUmZkV0UvZUxxQks3Q1lFbGFxYURtQVc4WlltZmZt?=
 =?utf-8?B?Z1RsRTZPcGVjTjhuQTcxbHpoTW1VSWhkcloyT2xMM2dqZjdhb0ZTRnJQbVIv?=
 =?utf-8?B?ZGN4eENlRGtLdGFXMS9sVG9kRG4wSDVRbFVzZGpqUHVCbVlZVnIzQU1Bc0dZ?=
 =?utf-8?B?R0hNYVgvc2p3N2ExQS94NnNuNVVNMFNRK3JQNXRwek1pNTM2R1pSZHVEdlV5?=
 =?utf-8?B?R1lmTmJMVjMyVUJUY0FkSlQ4NjZQSFpCRFBsL0tncENvdWNOQXR0djY0Nk5q?=
 =?utf-8?B?Vm9xeFFsWHRhRWY4MGw1dG13QjFvQTJ5dDBtd2VodlQ5ckFYYThacWF6K3Fo?=
 =?utf-8?B?eVR6TnhQZmVuVGYvZkNEWUNXVHZtelppZjV3MHVZcTYyczl3WmpLTENBUHFE?=
 =?utf-8?B?YTB0bDN5N3Y2UDJkSTMwT0FzaTdyL2FIRmwvVUxvWkI3RWRNVnV2UnJ6Wlpv?=
 =?utf-8?B?L1BCYmdvRkZPTDFPRFlhTGdDNG1FUEZSYXpvR3luZWU1TGVVY3dCMXF1d3U2?=
 =?utf-8?B?aHNHZDJEMEx2djdkWDdsalBuWUV3blBLWjcrTDhtMG9FYitmNmVUczhMNVNN?=
 =?utf-8?B?SlRFL3UxMjBWaTBjR0NtSURYU0VCZ01Tek9jaCs1bnJ6S1pvRTgzRFN3Vml3?=
 =?utf-8?B?ejNZZmhhc20yeTh2WlhRcXVXL1RZSG8vYktCZjZvRjhJL0dUZDQ2aDVQV2Nw?=
 =?utf-8?B?SjlJN2tGMUExZnprN1V0R29TZEphUXhSbk1NclR3Nm9jTmw0UTFHaDFCTmRY?=
 =?utf-8?B?dndOZjFDQ1RsUDFqTTBISFdQL1d0S2NXVmpxTUY2VTA1QW1udUdIaXo3cTRw?=
 =?utf-8?B?N3pRcUUwSjZSL3VzdWZPOE10bklNYTEvRlVtcWZKc3FxOXNLZ1VJLzFJc013?=
 =?utf-8?B?TXBHNHlNVTZtakY2ZllkdE5NTnpZbjMzT295OUpQZnZzc21LSmdzWGtLUFNM?=
 =?utf-8?B?UkUrc3VtcGlQN1UwVzQ5ZHhhcU5Rb2lVSjFpZ0hFVWZ0SGt6aGFWaHNUekJT?=
 =?utf-8?Q?ZGid24KGsx9YCbDaje2yiYQptU50Pk+qk1d0m4xtW4=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: ed1a3b02-dd97-4f42-df2c-08d9b904c769
X-MS-Exchange-CrossTenant-AuthSource: BN0PR11MB5744.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2021 22:07:22.8617
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: lB7u51+8SaIVxhX0q3RWp5TbAe/6jr1sysRC8eB4BQ6i8zAFB3mUM7LcMqPJmBL8DO92MPe/OIMG2Wfv/zLEIOlzT38E9lRGKMc4vHhRT6M=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR11MB4148
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Jarkko,

On 12/4/2021 3:47 PM, Jarkko Sakkinen wrote:
> On Wed, Dec 01, 2021 at 11:23:22AM -0800, Reinette Chatre wrote:
>> The page reclaimer ensures availability of EPC pages across all
>> enclaves. In support of this it runs independently from the individual
>> enclaves in order to take locks from the different enclaves as it writes
>> pages to swap.
>>
>> When needing to load a page from swap an EPC page needs to be available for
>> its contents to be loaded into. Loading an existing enclave page from swap
>> does not reclaim EPC pages directly if none are available, instead the
>> reclaimer is woken when the available EPC pages are found to be below a
>> watermark.
>>
>> When iterating over a large number of pages in an oversubscribed
>> environment there is a race between the reclaimer woken up and EPC pages
>> reclaimed fast enough for the page operations to proceed.
>>
>> Instead of tuning the race between the page operations and the reclaimer
>> the page operations instead makes sure that there are EPC pages available.
>>
>> Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
> 
> Why this needs to be part of this patch set?

When pages are modified they are required to be in the EPC and thus 
potentially need to be loaded from swap. When needing to modify a large 
number of pages in an oversubscribed environment there is a problem with 
the reclaimer providing free EPC pages fast enough for all the page 
modification operations to proceed.

What that means is that if a user attempts to modify a large range of 
pages in an oversubscribed environment it is likely that the operation 
will fail to complete but instead it would result in partial success of 
as many pages as was on the free list. This is because the reclaimer may 
not run fast enough to free up sufficient EPC pages in a dynamic way.

This becomes complicated for user space. It could increase the priority 
of the reclaimer but that has been found to be insufficient*. There 
would still not be a guarantee that after one page modification call 
fails enough pages would have been freed up in support of a second page 
modification call.

With this change it would be ensured that when pages are being modified 
that there are sufficient EPC pages available to support the modifications.

Reinette

* The test that follows this patch was used to explore this scenario.