Received-SPF: pass (google.com: domain of linux-kernel+bounces-80384-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Message-ID: <7b53e155-2622-4acb-b7c9-d22e623e4cb3@intel.com>
Date: Mon, 26 Feb 2024 14:38:26 +1300
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v9 10/15] x86/sgx: Add EPC reclamation in cgroup
 try_charge()
To: Haitao Huang <haitao.huang@linux.intel.com>, "Mehta, Sohil"
	<sohil.mehta@intel.com>, "mingo@redhat.com" <mingo@redhat.com>,
	"jarkko@kernel.org" <jarkko@kernel.org>, "x86@kernel.org" <x86@kernel.org>,
	"dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
	"cgroups@vger.kernel.org" <cgroups@vger.kernel.org>, "hpa@zytor.com"
	<hpa@zytor.com>, "tim.c.chen@linux.intel.com" <tim.c.chen@linux.intel.com>,
	"linux-sgx@vger.kernel.org" <linux-sgx@vger.kernel.org>, "mkoutny@suse.com"
	<mkoutny@suse.com>, "tglx@linutronix.de" <tglx@linutronix.de>,
	"tj@kernel.org" <tj@kernel.org>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, "bp@alien8.de" <bp@alien8.de>
CC: "mikko.ylinen@linux.intel.com" <mikko.ylinen@linux.intel.com>,
	"seanjc@google.com" <seanjc@google.com>, "anakrish@microsoft.com"
	<anakrish@microsoft.com>, "Zhang, Bo" <zhanb@microsoft.com>,
	"kristen@linux.intel.com" <kristen@linux.intel.com>, "yangjie@microsoft.com"
	<yangjie@microsoft.com>, "Li, Zhiquan1" <zhiquan1.li@intel.com>,
	"chrisyan@microsoft.com" <chrisyan@microsoft.com>
References: <20240205210638.157741-1-haitao.huang@linux.intel.com>
 <20240205210638.157741-11-haitao.huang@linux.intel.com>
 <c5d03171473821ebc9cb79e3dad4d1bf0074e674.camel@intel.com>
 <op.2jjzaqdwwjvjmi@hhuan26-mobl.amr.corp.intel.com>
 <4db8493b-35a2-474f-997c-5e6ac1b8bd11@intel.com>
 <op.2jkfeezjwjvjmi@hhuan26-mobl.amr.corp.intel.com>
 <c913193c0560c4372d2fdb31e9edb28bcb419f50.camel@intel.com>
 <op.2jlti6g9wjvjmi@hhuan26-mobl.amr.corp.intel.com>
Content-Language: en-US
From: "Huang, Kai" <kai.huang@intel.com>
In-Reply-To: <op.2jlti6g9wjvjmi@hhuan26-mobl.amr.corp.intel.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Q1RrRi9oMml1cFgwRUt5QlZaNDRFWU9HYVdVamxjZnd3TmZqejNWNVZuaGtO?=
 =?utf-8?B?eDZpa0QxZHY1VVU4bDlSaGd5Y01sSFRsaWsvZ29Oa2FDclAxZTRzc0V3TERX?=
 =?utf-8?B?dDNJY2FNUENOeWtTcGdPWUNPdUF1YnV4eFdDVVFPVHJpZ1BiSGplOEp2bWk0?=
 =?utf-8?B?cUh4UWF6bHJ4ZTlCUTRqNzlVUHBza3hVYzRSaHJYMnB2MWNpdyt3UTZZbXUz?=
 =?utf-8?B?eHR2dGtqQU9KRlJHRE9MZFRtVXE0T20rNmxNaHYwOVVsZlM0YUlSNXd0eXl6?=
 =?utf-8?B?ZGk1N1dNZ25rSng0VzlDbWNBRTNrMVdUVWE5TTRhblhPNERweHBIWUhReGRX?=
 =?utf-8?B?SWFhb0ZJVXV1R1JkZVJuZ1h6eVZFZFhhK3lKQyt0ODBLVTZTRnkrK3ZhV0Ew?=
 =?utf-8?B?NWxkd2JqWE4rcVMzdEtaSVUwRzZkYVF6dktyNzNRdVU5c0xldSt4VlVtb0dE?=
 =?utf-8?B?Y0MvTkJNQjg1Z3ExN3dKRU1rN0puSEZSQ0NxZHA3aTNDdVp3L0JEMHZXVUk2?=
 =?utf-8?B?N1VmRlVyTmhkMUFiS1lqaWM0Nk9EV2ZxWDRDdWJvZktWdE9teXdKdTA5TXhX?=
 =?utf-8?B?andDcG1EQkkzdnBuSkhuVzNEdDlCYlVpMHRiaGVUODUzYytPdlpNK2Q2S21y?=
 =?utf-8?B?VTNZakhwWnNLQzNRdnJzdlZXczk5UzZXNmd0OHhORWVPd3M1RzkyMzlhOEFP?=
 =?utf-8?B?eSs4TEJmNGVQVkFBbXFnRld5cFRCeVQwYklhMnM1WlBTK1hRRkljUEwrMlhG?=
 =?utf-8?B?Ti9wNy8weDVBRTNkUWduT1JiclVmQ1lQbHNqa0I1cUFIanJFNmNYWG0xSStk?=
 =?utf-8?B?RkVXRUhHSzRaQ0F2dVdwdEduMkgwRktjMFNFRVl1RmFMV3pBSi9uN09xQW5M?=
 =?utf-8?B?ODR5R216YVp3eC9VczNsM1k4aUZud1BKTDZXbnIrSmsvVEEvV1YwaEZoSG1N?=
 =?utf-8?B?dkt5VGk3MUIwc1JFUVNDZVhCV2VheXdvTlhDQVZxY1hDVnB5NXpsMlJEZTZO?=
 =?utf-8?B?Qm40b2t0Yk9YbTVCb3g0cklEcnpQVEdleGYwV0ZPaHd0YXB3K2drNmxpRHFn?=
 =?utf-8?B?T2g3bExJbjViQkVuNVFUL2JxOGhWYkFPZHdXN3ViZGRkZDZzdnJ2cVViN2ZW?=
 =?utf-8?B?RUtXcDNsRWtaVy9mMHRrL3dIZXpZckdCMXl2QmxGYldUeHRZbnhSWnArNDM0?=
 =?utf-8?B?Z09RTDVLZCtmUnl0T3BmNERROVUwaFdDc2NCYXVFZzBNVkFRdC9jTzl3Mlky?=
 =?utf-8?B?TXFmS0Jna2tXOTVxdTZsL2lBTHgvSVg5bHI2TmM3T3J5WHMwWHlkZiswWnpq?=
 =?utf-8?B?T3hKWXgxR29KdU85TGVMaERSZmNyU3lpRG1SZ1k1N0RqY3VzdnNFNW40dldr?=
 =?utf-8?B?ZUxmUnEwMytRWXdaZG1HUVlybnZNYVBCMElYbitlWGx0SFlhM0MvU1NlaUEr?=
 =?utf-8?B?NEE0QW1tc2VpRzBsd20wTC93NUNsbng5UVVjRnl4cGlsOWQ5bFdaUXE1aWVC?=
 =?utf-8?B?OVdRYXA4blJHbko0Z1FFbXZDUW1jV2UwWkJTelRoWVluQ3I3emhMNVN1YmUx?=
 =?utf-8?B?VGJFN1Q3OS9GMzZQNWRuYUVTQWczamJJZDkvL3ozckd6akdVMFJpWFEwL1dt?=
 =?utf-8?B?VHJWanF1dEcraUV5b3FwbUdYTDFtc3pzTkxGL2ppemNENGIrcUkvckhodzdF?=
 =?utf-8?B?Mm93clZXMUoybXd4Y3BXYTN3TGdydHVFOWs1Z3lZVFk2eTlVL2pDOC85Qm0y?=
 =?utf-8?B?NFV2NWxtU2dtMTV3UDJ3Z1RXc29jbkVqaktKbXcvWnVZM29rdGhNWWg0WWJa?=
 =?utf-8?B?SjFOZnpxRUFMVkswZ0MxZmFIWjdjMnY5OFNaWFBDUFpLek1iV28yeG5FSGw0?=
 =?utf-8?B?eXptR2NqZzdLS0s1RzVNVlVDQnZ2azcrRE15WFpMMmNOMnpoZU96TDArTXRp?=
 =?utf-8?B?blNmSmh6czdXTDJtajE4ekx1azQwL1ZQRTZyT2Q2WDU0UnRXNzFZUnBhQy9r?=
 =?utf-8?B?Y2tJSzdJc1lzUmRyK3VXZUVHR21CMnZmWkRSdUs4REc4NnErYURQQi91cHNM?=
 =?utf-8?B?SWJHbk01Z0JXbG9Pcll5cERTTU9tMGJDWS9XTkJmTlZONGZjUkUzWTNoSVFa?=
 =?utf-8?Q?juG+t3/YpOVvjq7Oe3Aqf6YdL?=
X-MS-Exchange-CrossTenant-Network-Message-Id: e8cddcc2-7082-4110-7dc9-08dc366bad20
X-MS-Exchange-CrossTenant-AuthSource: BL1PR11MB5978.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Feb 2024 01:38:47.5369
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: m6bhsmxRYsZcHHjysOXrsSBmrFlGJODD+H4nsBLlJVRdIrxmExdMR4lAZAGUqpKeNTwbgoSixDidGwujAy1+XQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR11MB5546
X-OriginatorOrg: intel.com


On 24/02/2024 6:00 am, Haitao Huang wrote:
> On Fri, 23 Feb 2024 04:18:18 -0600, Huang, Kai <kai.huang@intel.com> wrote:
> 
>>> >
>>> Right. When code reaches to here, we already passed reclaim per cgroup.
>>
>> Yes if try_charge() failed we must do pre-cgroup reclaim.
>>
>>> The cgroup may not at or reach limit but system has run out of physical
>>> EPC.
>>>
>>
>> But after try_charge() we can still choose to reclaim from the current 
>> group,
>> but not necessarily have to be global, right?  I am not sure whether I am
>> missing something, but could you elaborate why we should choose to 
>> reclaim from
>> the global?
>>
> 
> Once try_charge is done and returns zero that means the cgroup usage is 
> charged and it's not over usage limit. So you really can't reclaim from 
> that cgroup if allocation failed. The only  thing you can do is to 
> reclaim globally.

Sorry I still cannot establish the logic here.

Let's say the sum of all cgroups are greater than the physical EPC, and 
elclave(s) in each cgroup could potentially fault w/o reaching cgroup's 
limit.

In this case, when enclave(s) in one cgroup faults, why we cannot 
reclaim from the current cgroup, but have to reclaim from global?

Is there any real downside of the former, or you just want to follow the 
reclaim logic w/o cgroup at all?

IIUC, there's at least one advantage of reclaim from the current group, 
that faults of enclave(s) in one group won't impact other enclaves in 
other cgroups.  E.g., in this way other enclaves in other groups may 
never need to trigger faults.

Or perhaps I am missing anything?