Received-SPF: pass (google.com: domain of linux-kernel+bounces-77401-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <4db8493b-35a2-474f-997c-5e6ac1b8bd11@intel.com>
Date: Fri, 23 Feb 2024 10:26:05 +1300
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v9 10/15] x86/sgx: Add EPC reclamation in cgroup
 try_charge()
Content-Language: en-US
To: Haitao Huang <haitao.huang@linux.intel.com>, "hpa@zytor.com"
	<hpa@zytor.com>, "tim.c.chen@linux.intel.com" <tim.c.chen@linux.intel.com>,
	"linux-sgx@vger.kernel.org" <linux-sgx@vger.kernel.org>, "x86@kernel.org"
	<x86@kernel.org>, "dave.hansen@linux.intel.com"
	<dave.hansen@linux.intel.com>, "jarkko@kernel.org" <jarkko@kernel.org>,
	"cgroups@vger.kernel.org" <cgroups@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"mkoutny@suse.com" <mkoutny@suse.com>, "tglx@linutronix.de"
	<tglx@linutronix.de>, "Mehta, Sohil" <sohil.mehta@intel.com>, "tj@kernel.org"
	<tj@kernel.org>, "mingo@redhat.com" <mingo@redhat.com>, "bp@alien8.de"
	<bp@alien8.de>
CC: "mikko.ylinen@linux.intel.com" <mikko.ylinen@linux.intel.com>,
	"seanjc@google.com" <seanjc@google.com>, "anakrish@microsoft.com"
	<anakrish@microsoft.com>, "Zhang, Bo" <zhanb@microsoft.com>,
	"kristen@linux.intel.com" <kristen@linux.intel.com>, "yangjie@microsoft.com"
	<yangjie@microsoft.com>, "Li, Zhiquan1" <zhiquan1.li@intel.com>,
	"chrisyan@microsoft.com" <chrisyan@microsoft.com>
References: <20240205210638.157741-1-haitao.huang@linux.intel.com>
 <20240205210638.157741-11-haitao.huang@linux.intel.com>
 <c5d03171473821ebc9cb79e3dad4d1bf0074e674.camel@intel.com>
 <op.2jjzaqdwwjvjmi@hhuan26-mobl.amr.corp.intel.com>
From: "Huang, Kai" <kai.huang@intel.com>
In-Reply-To: <op.2jjzaqdwwjvjmi@hhuan26-mobl.amr.corp.intel.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dDhaTDM0WTllYzZxNjRrc24xM3p3OGVtSm0ra0lmSm9hcVRoY2JMZ013V3lm?=
 =?utf-8?B?WE0zM0k1S0RaazJadkcxTkYwdTgrbmpJYmJJVG5UbTRCSGV3bWpyeVFRcGlw?=
 =?utf-8?B?Q1hkeTlYUTRBVVVHWVIreFl2eUZhcGY0VDJycXozUFR6OXdxSVhFdlRqUU1L?=
 =?utf-8?B?TDBidFQ5YUVpWlZuVnZYQlQwNnR6dStpU1oxa2g1cEpVS09Dd2RnVzVCd2FN?=
 =?utf-8?B?ZmVYbjhBbU9mY2VuNlF2eUk5aHE5bTNVS1FyTyt4VEJBbFdaaFFaaEdXZ1V6?=
 =?utf-8?B?bkI5MGpaeGxWYzlmK3RyejUrRDNRODV0Zkd6NVRrRHU1cTBOOU44WEYzQ2V3?=
 =?utf-8?B?THh5VUxCVnNoVWNkd3JZYjFzMGtSRUJZcVdKOVNCYzBYWFI5NTJXOVRpOUg5?=
 =?utf-8?B?cEY3WVFlK2tBbmRJRXlhbnQ2T2RsV1B6dU5kR0FicmREM0hGR3JmcWd1YlJ6?=
 =?utf-8?B?bTA5VHBUUjRxQTRCTG51ME9YNTZHMWZhNzgrSEl5UUFtV3FDTnk2bU9wNkFi?=
 =?utf-8?B?TVhVM0xycEJyWFVVZkJwVHdpUVRxb1ZnOUNhWlRlUjF5L09uVzBCbTZSZnJl?=
 =?utf-8?B?emNOUGt6TW90VUg5TzdXUnpKc0dzNGlNRmhtWkU4UEE3blVQRklaYzU2VGpY?=
 =?utf-8?B?NUViSm5raDU2THFYWlc1aDJJNWp1Y0lidUk2OWhja0VGUHYvYzUzUmtFYXZY?=
 =?utf-8?B?WkNFN2R3U291YlpPdXkxNEZtMXZCUnh0eWJDSFkrMVcrNVE4TUpPZWp6SVll?=
 =?utf-8?B?RmsrdzdYQ1JTd21wZVd3aG9sb0ZMTHZnNWd1UkVpM1N3dFVqNG16L0p5bGRT?=
 =?utf-8?B?Q1dpa0FNZEhvMHo5dm5obE5XNlRWYUIwMjlIMEUvSHhJcUpXMjBCUHUyMEJp?=
 =?utf-8?B?VWovQk9CWmtRL2RsUEJsTmdCRzU3amRSejJTMUFscnpKbmFiQnJsQ21CajhF?=
 =?utf-8?B?MHhtcWR5MVgvVnc2TG5ISHNpUGxNMUoxSDMwdWMxS0RoRTZzakV5bW5sakp6?=
 =?utf-8?B?amFaZzd4N3hMcHErWlloWkt1UnVLWE5yb0FNaExhSis0ME5ueUJYeG1QY2Ix?=
 =?utf-8?B?OGVveDlFL00yNlRaY2FZOG5QamRKVUs3c281amVpUGpiWllabDdqS0VBYU1C?=
 =?utf-8?B?QUQ1S0Fvc01LaU9sUlRickcxNmZ4TWdCLzFIYVA3OFpJeTVFaHF2VEZSSGUv?=
 =?utf-8?B?VlhMWEZ4TlJta1lOb2pxQVhjRUVrclV3VG8zM2E3c3ZwNjU2RVJvUzZpa3o4?=
 =?utf-8?B?b2F0YlZEL1k4d0c5ZlpaYS9XRm9OTWxKa0wvWE5NWFgrSDRrWkU2YW9vOGd2?=
 =?utf-8?B?MHZCRGxNdDY3ZnN6bGVJQWNERytwSnljOHBTRFNiUE1EZllqQVNLZFY5UkpH?=
 =?utf-8?B?SVlvZFNueFZ3djF6NEg0TnpZdmllQnAzMXBLUDJzcFkwdTcwNVRvVlAyQ3F3?=
 =?utf-8?B?SC9zY2Z3c0FleFNUNkJpeVZER0ROMnZNMWwxSmtSYjIxUHQrVlV6emZOakVo?=
 =?utf-8?B?VTZqVUQrenB2Qm1xUmxUalduUjNLVVlTc3czYTR0d3N2YjREa0p1WXpjTFFX?=
 =?utf-8?B?U3NNTDk0eXJCZExYVXFOUC9CRmhuRkI4MzNZeUdobDRucW0xbGs4eUQyTlho?=
 =?utf-8?B?UTJMc2grNmdydStlRUdIZ1ZYV1FvaDgySmFNUW5CdENzTnUyQVRtay9BTnMw?=
 =?utf-8?B?cGdWSFQ2elNkYmRMQis1RVZUcTN1SERXMitiNWVXdjRhREVqNE1ENDNvdXVG?=
 =?utf-8?B?eEFORjRKcS9lbjdkQnNVeVE0ektnZVlwVll4NTVYaGp2a1g1QU8xSXpjVG5C?=
 =?utf-8?B?MHRPb0N1STlJU0IxVmV0dFFmTjB2R1IvM1o0MjRjMmUycFk5dXIvWDMvN0NL?=
 =?utf-8?B?R2c2WEhDOTRiWlFMWXNIa1BvTHN4dVJ6Vm0xREx0R1FCbU1BdVlSRGJyTWVV?=
 =?utf-8?B?aDBDUW00Q1lsZnZtSm84ZDg4NlhzVXVmMVJqa1lOV2ZibE01MVRxRTc0Wmsy?=
 =?utf-8?B?cTU3dFQ2QVFRTkdYWGpWQ3pEbU55enZYZjdGZDBXUnpCdWdCZWdndHpxWk1s?=
 =?utf-8?B?R2dPR2xabkFxS3FMM1l6QmNZWE1IUk5lTjNlamxtYVpVL1dUUGZ6KytRenRP?=
 =?utf-8?Q?/fGf9eGO9dzYxQIJf/GInugf3?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 54923eac-8c89-4268-ede2-08dc33ece8e7
X-MS-Exchange-CrossTenant-AuthSource: BL1PR11MB5978.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Feb 2024 21:26:19.4731
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: pa+Fq12wQDtFgM8fNEwh4C2YSJQ3K1VFVZbpjj2ZR3q+eSImHyOOfVIe2E5vfvTC1ZbomRZqKH9/1Pvx3ZXeXA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR11MB5355
X-OriginatorOrg: intel.com


On 23/02/2024 6:09 am, Haitao Huang wrote:
> On Wed, 21 Feb 2024 05:06:02 -0600, Huang, Kai <kai.huang@intel.com> wrote:
> 
>>
>>> -int sgx_epc_cgroup_try_charge(struct sgx_epc_cgroup *epc_cg)
>>> +int sgx_epc_cgroup_try_charge(struct sgx_epc_cgroup *epc_cg, bool 
>>> reclaim)
>>>  {
>>> -    return misc_cg_try_charge(MISC_CG_RES_SGX_EPC, epc_cg->cg, 
>>> PAGE_SIZE);
>>> +    for (;;) {
>>> +        if (!misc_cg_try_charge(MISC_CG_RES_SGX_EPC, epc_cg->cg,
>>> +                    PAGE_SIZE))
>>> +            break;
>>> +
>>> +        if (sgx_epc_cgroup_lru_empty(epc_cg->cg))
>>> +            return -ENOMEM;
>>> +
>>> +        if (signal_pending(current))
>>> +            return -ERESTARTSYS;
>>> +
>>> +        if (!reclaim) {
>>> +            queue_work(sgx_epc_cg_wq, &epc_cg->reclaim_work);
>>> +            return -EBUSY;
>>> +        }
>>> +
>>> +        if (!sgx_epc_cgroup_reclaim_pages(epc_cg->cg, false))
>>> +            /* All pages were too young to reclaim, try again a 
>>> little later */
>>> +            schedule();
>>> +    }
>>> +
>>> +    return 0;
>>>  }
>>>
>>
>> Seems this code change is 90% similar to the existing code in the
>> sgx_alloc_epc_page():
>>
>>     ...
>>     for ( ; ; ) {
>>                 page = __sgx_alloc_epc_page();
>>                 if (!IS_ERR(page)) {
>>                         page->owner = owner;
>>                         break;
>>                 }
>>
>>                 if (list_empty(&sgx_active_page_list))
>>                         return ERR_PTR(-ENOMEM);
>>
>>                 if (!reclaim) {
>>                         page = ERR_PTR(-EBUSY);
>>                         break;
>>                 }
>>
>>                 if (signal_pending(current)) {
>>                         page = ERR_PTR(-ERESTARTSYS);
>>                         break;
>>                 }
>>
>>                 sgx_reclaim_pages();
>>                 cond_resched();
>>         }
>>     ...
>>
>> Is it better to move the logic/code change in try_charge() out to
>> sgx_alloc_epc_page() to unify them?
>>
>> IIUC, the logic is quite similar: When you either failed to allocate 
>> one page,
>> or failed to charge one page, you try to reclaim EPC page(s) from the 
>> current
>> EPC cgroup, either directly or indirectly.
>>
>> No?
> 
> Only these lines are the same:
>                  if (!reclaim) {
>                          page = ERR_PTR(-EBUSY);
>                          break;
>                  }
> 
>                  if (signal_pending(current)) {
>                          page = ERR_PTR(-ERESTARTSYS);
>                          break;
>                  }
> 
> In sgx_alloc_epc_page() we do global reclamation but here we do 
> per-cgroup reclamation. 

But why?  If we failed to allocate, shouldn't we try to reclaim from the 
_current_ EPC cgroup instead of global?  E.g., I thought one enclave in 
one EPC cgroup requesting insane amount of EPC shouldn't impact enclaves 
inside other cgroups?

That's why the logic of other lines is different
> though they look similar due to similar function names. For the global 
> reclamation we need consider case in that cgroup is not enabled. 
> Similarly list_empty(&sgx_active_page_list) would have to be changed to 
> check root cgroup if cgroups enabled otherwise check global LRU.  The 
> (!reclaim) case is also different.  

W/o getting clear on my above question, so far I am not convinced why 
such difference cannot be hide inside wrapper function(s).

So I don't see an obvious good way
> to abstract those to get meaningful savings.
> 
> Thanks
> Haitao