Received-SPF: pass (google.com: domain of linux-kernel+bounces-150903-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <640866c5-9fe0-4f7b-a459-7a685dbe4092@intel.com>
Date: Fri, 19 Apr 2024 13:32:14 +1200
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v12 09/14] x86/sgx: Implement async reclamation for cgroup
References: <20240416032011.58578-1-haitao.huang@linux.intel.com>
 <20240416032011.58578-10-haitao.huang@linux.intel.com>
Content-Language: en-US
From: "Huang, Kai" <kai.huang@intel.com>
To: Haitao Huang <haitao.huang@linux.intel.com>, <jarkko@kernel.org>,
	<dave.hansen@linux.intel.com>, <tj@kernel.org>, <mkoutny@suse.com>,
	<linux-kernel@vger.kernel.org>, <linux-sgx@vger.kernel.org>,
	<x86@kernel.org>, <cgroups@vger.kernel.org>, <tglx@linutronix.de>,
	<mingo@redhat.com>, <bp@alien8.de>, <hpa@zytor.com>, <sohil.mehta@intel.com>,
	<tim.c.chen@linux.intel.com>
CC: <zhiquan1.li@intel.com>, <kristen@linux.intel.com>, <seanjc@google.com>,
	<zhanb@microsoft.com>, <anakrish@microsoft.com>,
	<mikko.ylinen@linux.intel.com>, <yangjie@microsoft.com>,
	<chrisyan@microsoft.com>
In-Reply-To: <20240416032011.58578-10-haitao.huang@linux.intel.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?aEFpaEhseTQydmw5dERGL1h1d0FRdVFZM2tjNlNCRkpjMGFaUTZKNEhuTDFZ?=
 =?utf-8?B?Rk5IZDB3TVVwL1NXd2N4eGZseVFIZjR4R0JiOXUwbW8xdnYrZnloRzFRMWw1?=
 =?utf-8?B?Mjl3c2hSM0ZrdVRtY3k2S2lUdHBnRXJiUWNnNTJVRVJpc3NNS2RjM1M5Nzhk?=
 =?utf-8?B?aUVaallxNEYwWUR0OFkxMGo4aXMzTlFhSTRhQ1ZvVGNoQ0xmMW82Ym9hZWFD?=
 =?utf-8?B?ZlRyRStGT1I2enJ5cmUwL0tYUUZmRzZ6ZUQ2eTFXR3BsRGJnMW1WOHFJdWpD?=
 =?utf-8?B?T1cyYlhQb3FPU0FMbWUvZjF1SDJscHFOeldEODd1Z09ob0s4REptYmltaVFS?=
 =?utf-8?B?YlhzR0Rwakt4V1dqZ01zZFlFVUpaZ2FUcjNRZDNEdzd4OUN4Wk1LL1BaSktF?=
 =?utf-8?B?cm1SMThCa1hndW5ydEZhMEcrTjFpY09OL052MS9hRUhqWEpiVmZrS0VtRWV1?=
 =?utf-8?B?M3E4VkZiRkpvZzRVRVJRMlRBTEFjcEFuSjEvZUJGckp2aWswbUVtQUZ1RHBS?=
 =?utf-8?B?aU0xVjZTSTFZOVFkUktYSWlaUnYzQ1hmU3prQlN6a2dIRVVlVnhUcTVSbWxR?=
 =?utf-8?B?UlQzc2FXcS94Zmhhek4xVjR2WVdrRDhiS0JDMzhZYUpTWXhZK1c3Q1VqVFNS?=
 =?utf-8?B?ZkFoNlY3eGRuWVIrZ2FLUjY0RzlkVHVVN0hTQkw0VktrbzRuY1RhZG9vcG5B?=
 =?utf-8?B?bHUwYmQxRWl5ZFE1V0Y4OFQycHdzMk5Zd2xXWDdzbmN5cm5iRFhCYWM3dFls?=
 =?utf-8?B?Y3NQMERzL1VKbXFVcHdELzdJU0pNR2NIWWZJWTdXQk5Zb0JBdmVUeWIxVzE1?=
 =?utf-8?B?WUxQUzREa1dYT0twMnI5bkRqd2tYQWx0TzJxdWphWVZSa3lDQWU1ZmdyZ211?=
 =?utf-8?B?RFpuTUt6djJoQzNTTm9tdFBZemFnemNLMTB0V1ZYVU53RFhiT0ZsbTRlVXpj?=
 =?utf-8?B?Q25kU2dMcDErTXIySnNBcWZIL0s0M2RRZDN3OFJxVHA3VEZTTXhqYjdGa0VN?=
 =?utf-8?B?d1hLSEdDc0NVTDRtVFlrQUpQY1IydVVlMWZIUXE0cWJxNXNjck54alpmMnp0?=
 =?utf-8?B?MTd0MUxqVkJmUVFUMFZlY2dOUG53UTRzVDArWm5rb0dqTjI5NHdsU05qYWV3?=
 =?utf-8?B?TjhqUmpVTDR4MFlqRU5iWnJvbjlPZnN2bE5hUGlFMmRpek9zaDYrTWdjWmkr?=
 =?utf-8?B?TmsyMTBCWEVxMG9OcDFFcGlGQk94cWJRbThtUVJPNTliaHZ6VWUxdHQvL2tv?=
 =?utf-8?B?K1E1MXZBQUVsZmhJbUlKdFpwNXFlSFFLczFXN21QMWMxaEdLMTBMY3ZPMmda?=
 =?utf-8?B?UW14b042eHNDL2I1bGk5S0ttUVVMOTVWTkNuazJDdTNDd0s1RTI0RkR0bW9i?=
 =?utf-8?B?MUtISDZNVThBbDdXQXZnbEV1U0w4UXlpdzl2L2RaSmJEcm5nWlgvdUk3SEdx?=
 =?utf-8?B?bGdwQXc1OFF4MzdlWTV1dFBFMjV3eVp5WVBpU093b3lDRVU5dFJYeks5Rm5z?=
 =?utf-8?B?UlE3ODNxMVlzdEEzSFlJc0VhMGJrei9aR1ZWTkhrYTBxQzByd2R4YWVzaWVN?=
 =?utf-8?B?TWlHWEdTT3pXQVlXUVNNWmtGSmhOZWI4UTJmSERTZ0MvMWpRVVZhZklwMTVN?=
 =?utf-8?B?NXhiUTdsZ214OHNKMmg5aFFuUDlFNFJoNnhpbmVhb01kWVBNS1MwdUErMi9E?=
 =?utf-8?B?RjFWaGhmZGdpbnFrR1J1aFQwYjZNU0FlcDh0REFRQllKWms2YnBUYTFrRnZH?=
 =?utf-8?B?U21FclRJSENXQ2o2LzEvcysvdzdNWDMvcTdHcDJlNm9DU3FJWHRPbVFkeG10?=
 =?utf-8?B?ZHRTTkczaTFGNmF6M0VQcmNMcDlQN0JBMFpPaXdWRjZBVjhzajc0Q08yNHJw?=
 =?utf-8?B?bjFPV3VwTG9aNGNUNi9qRHN0VEZzSGZoRWxJb3RpeHI1b2lWeU5GekplaWpi?=
 =?utf-8?B?cWdBR2lsNzc3MGRzWUFYdFNWY0gxMHJQRCtXSzJ5Zzd3c2M3V3h6dkcyeW9C?=
 =?utf-8?B?UzJZMXl6VVRPc3lleUI3S2xxYy9hTk9mSzNDZm5mL0VnK04xWFZ6bTlIMEFT?=
 =?utf-8?B?R0JjYk83Q1psZ2dWY0NldDBUUjJKRmtDMENWQXFjeDVCenBaeVFqa3lNYlRj?=
 =?utf-8?Q?5giyPpqcPdpL2hRJGTS9PjPcc?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 6473134e-03e8-40eb-739b-08dc60109215
X-MS-Exchange-CrossTenant-AuthSource: BL1PR11MB5978.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Apr 2024 01:32:26.8036
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: U64ZlP+YB9YFcv3whQl0lyfvlaJJkRsZRfJtQOZHlJU2TyBaitHepJWH9+53YUgne7kN8O4lTkKLAW7S6KkfDg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR11MB6222
X-OriginatorOrg: intel.com


On 16/04/2024 3:20 pm, Haitao Huang wrote:
> From: Kristen Carlson Accardi <kristen@linux.intel.com>
> 
> In cases EPC pages need be allocated during a page fault and the cgroup
> usage is near its limit, an asynchronous reclamation needs be triggered
> to avoid blocking the page fault handling.
> 
> Create a workqueue, corresponding work item and function definitions
> for EPC cgroup to support the asynchronous reclamation.
> 
> In case the workqueue allocation is failed during init, disable cgroup.

It's fine and reasonable to disable (SGX EPC) cgroup.  The problem is 
"exactly what does this mean" isn't quite clear.

Given SGX EPC is just one type of MISC cgroup resources, we cannot just 
disable MISC cgroup as a whole.

So, the first interpretation is we treat the entire MISC_CG_RES_SGX 
resource type doesn't exist, that is, we just don't show control files 
in the file system, and all EPC pages are tracked in the global list.

But it might be not straightforward to implement in the SGX driver, 
i.e., we might need to do more MISC cgroup core code change to make it 
being able to support disable particular resource at runtime -- I need 
to double check.

So if that is not something worth to do, we will still need to live with 
the fact that, the user is still able to create SGX cgroup in the 
hierarchy and see those control files, and being able to read/write them.

The second interpretation I suppose is, although the SGX cgroup is still 
seen as supported in userspace, in kernel we just treat it doesn't exist.

Specifically, that means: 1) we always return the root SGX cgroup for 
any EPC page when allocating a new one; 2) as a result, we still track 
all EPC pages in a single global list.

But from the code below ...


>   static int __sgx_cgroup_try_charge(struct sgx_cgroup *epc_cg)
>   {
>   	if (!misc_cg_try_charge(MISC_CG_RES_SGX_EPC, epc_cg->cg, PAGE_SIZE))
> @@ -117,19 +226,28 @@ int sgx_cgroup_try_charge(struct sgx_cgroup *sgx_cg, enum sgx_reclaim reclaim)
>   {
>   	int ret;
>   
> +	/* cgroup disabled due to wq allocation failure during sgx_cgroup_init(). */
> +	if (!sgx_cg_wq)
> +		return 0;
> +

.., IIUC you choose a (third) solution that is even one more step back:

It just makes try_charge() always succeed, but EPC pages are still 
managed in the "per-cgroup" list.

But this solution, AFAICT, doesn't work.  The reason is when you fail to 
allocate EPC page you will do the global reclaim, but now the global 
list is empty.

Am I missing anything?

So my thinking is, we have two options:

1) Modify the MISC cgroup core code to allow the kernel to disable one 
particular resource.  It shouldn't be hard, e.g., we can add a 
'disabled' flag to the 'struct misc_res'.

Hmm.. wait, after checking, the MISC cgroup won't show any control files 
if the "capacity" of the resource is 0:

"
  * Miscellaneous resources capacity for the entire machine. 0 capacity
  * means resource is not initialized or not present in the host.
"

So I really suppose we should go with this route, i.e., by just setting 
the EPC capacity to 0?

Note misc_cg_try_charge() will fail if capacity is 0, but we can make it 
return success by explicitly check whether SGX cgroup is disabled by 
using a helper, e.g., sgx_cgroup_disabled().

And you always return the root SGX cgroup in sgx_get_current_cg() when 
sgx_cgroup_disabled() is true.

And in sgx_reclaim_pages_global(), you do something like:

	static void sgx_reclaim_pages_global(..)
	{
	#ifdef CONFIG_CGROUP_MISC
		if (sgx_cgroup_disabled())
			sgx_reclaim_pages(&sgx_root_cg.lru);
		else
			sgx_cgroup_reclaim_pages(misc_cg_root());
	#else
		sgx_reclaim_pages(&sgx_global_list);
	#endif
	}

I am perhaps missing some other spots too but you got the idea.

At last, after typing those, I believe we should have a separate patch 
to handle disable SGX cgroup at initialization time.  And you can even 
put this patch _somewhere_ after the patch

	"x86/sgx: Implement basic EPC misc cgroup functionality"

and before this patch.

It makes sense to have such patch anyway, because with it we can easily 
to add a kernel command line 'sgx_cgroup=disabled" if the user wants it 
disabled (when someone has such requirement in the future).