Received-SPF: pass (google.com: domain of linux-kernel+bounces-170518-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <966e4afe-b177-4527-80c9-1146d2503c93@intel.com>
Date: Tue, 7 May 2024 12:10:42 +1200
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v13 12/14] x86/sgx: Turn on per-cgroup EPC reclamation
To: Haitao Huang <haitao.huang@linux.intel.com>, <jarkko@kernel.org>,
	<dave.hansen@linux.intel.com>, <tj@kernel.org>, <mkoutny@suse.com>,
	<linux-kernel@vger.kernel.org>, <linux-sgx@vger.kernel.org>,
	<x86@kernel.org>, <cgroups@vger.kernel.org>, <tglx@linutronix.de>,
	<mingo@redhat.com>, <bp@alien8.de>, <hpa@zytor.com>, <sohil.mehta@intel.com>,
	<tim.c.chen@linux.intel.com>
CC: <zhiquan1.li@intel.com>, <kristen@linux.intel.com>, <seanjc@google.com>,
	<zhanb@microsoft.com>, <anakrish@microsoft.com>,
	<mikko.ylinen@linux.intel.com>, <yangjie@microsoft.com>,
	<chrisyan@microsoft.com>
References: <20240430195108.5676-1-haitao.huang@linux.intel.com>
 <20240430195108.5676-13-haitao.huang@linux.intel.com>
Content-Language: en-US
From: "Huang, Kai" <kai.huang@intel.com>
In-Reply-To: <20240430195108.5676-13-haitao.huang@linux.intel.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?emZxaWlvZVhreG5rOVB3bU16a213eU5hbU1tUGxtRUZqOFNCWmNNcFczSnI2?=
 =?utf-8?B?WFlCajc3TzFieDZPR0Y5a05iWkgxcDAzeU9BWU5aR2tzVjVJNXJNMTFXa0hB?=
 =?utf-8?B?QkUwcElDYzV4elI1TGJObWlYM1hNaDRacTZzRThWSjVHVWJ1VEU1T2JZWXJv?=
 =?utf-8?B?ZzRBQVk5WEVuOE4wSDhEQjY5M3NtL2QzYThzZlVKUnlaUEx4MmVIanRUQlU3?=
 =?utf-8?B?aTBBTmprOUR1UFBPWXo2OUZKcUphZ21ZSmlMallITzdPa3RJdmhvNzcxOEEy?=
 =?utf-8?B?K0pYcStlelV2bDliZUpYK2FBWnFEWEVMbW9MQW5teWoxTEptMHFDeEI0NnFi?=
 =?utf-8?B?SUIwaGlVN0U2Ynkva2w5T3VsTlplcFJUTkhlSTlqK2pKR1MwZzExNEdXS1Jt?=
 =?utf-8?B?NGloOWhFaXY0VU53RjdPV1UrMEdoZTBRZnU5R3RTMlZRTE1ncFlMNUlTbmxk?=
 =?utf-8?B?TkdtU0FOYnBHdm53a09WMHV5dStXYlJtNjlMS2NrOVBpeFg4ZDdlcUUvaitN?=
 =?utf-8?B?bUZyU3Y5ZFNleEY1cXllcTAyOUdFV3I1V25VTzJXS3NIU1p3QkhRQkVyZEZQ?=
 =?utf-8?B?Rit1L0RsQjdRaHhIYU9XNkdqSUR2MEM1TGNUV2ZDTmp4Q3JYN202dEVqNEll?=
 =?utf-8?B?aHBFN1l0eGh4TTZNcXRxMUlHZ3VwdFB4MTBabjRLb0RESkx6N3pJU2VYOG9F?=
 =?utf-8?B?cDlyUzdWYkVnZ3BhM3dOaDZ6dlFqNHNaTWVpTmVuQjlBSGg5NW9VZVpzK2la?=
 =?utf-8?B?UzFFV0d0a1FQUUlPYkhQdXc3MU1lTEpGUVpSY3JoaDRZQkMyajhZYmpibjEv?=
 =?utf-8?B?VTh5U2c1WG1sZWZVbFN2am1Na2drakRkTSs0RVVsUXh6b3JnL1kxKzFqanJt?=
 =?utf-8?B?cTQ1cGVMYm05cXlVU3NnbjVFeXcyeGNwV0JIZnlPS1dDYkRvRVdDOGsvYWw3?=
 =?utf-8?B?dCtLakVrVkF6SkcvSG1oZDVNMmxCd0pKSUVrY2U2TWJ0OWs5M0Y4NDVRM2Iy?=
 =?utf-8?B?cFQ3aE5EQy9UV1loU1ZjSVdENDZ2Y0lqeVlZTW11TmZZaUFnZnJkcW43UjJq?=
 =?utf-8?B?czBsQ09QcHpIYTZwT2piOTNnMTJrclNwWHhFUFUvZ3BzWElrR0xUVkxaK2NK?=
 =?utf-8?B?SUtldGVjVGVHcmJKdURwUFhibUk3ZEhtaWZBTGEyaUVqK29PWWl3L1g4NGpO?=
 =?utf-8?B?aGsrbnJIOWZXYlJxS3dBVmhBb3pKWk82OWlVWFpiWFhyQ1laaTRnSzhGeUd1?=
 =?utf-8?B?TjcxYTlmTlZoNEVhWUxrZ3lPek8xNU5uNHptWitmNUhXTVY0WTR4aGNVTmhH?=
 =?utf-8?B?YlZWYnIzTDJzZGp4eHlJWUFGYWlqSkxIcmVyaElpaDJSODZjWmtCa2JGRTBj?=
 =?utf-8?B?SDdUSWxpRzZZOUlick4vaUdmZThCVWlzcHlOeUhKblF1K1dGU3FuV2xUZHdv?=
 =?utf-8?B?VXN6M1dtVUhERDNXSWZTUTQyUVpnNlRRYVl5RWlRTW1TUmQ3NnhwdGd0RWVC?=
 =?utf-8?B?QklQMURzNTlULzRPOEVDSHpOMzZCWEhtQ2NBR2luMUMwWHlCeWc5aUdOd3Jz?=
 =?utf-8?B?cXZmSTlEdmJhOGdLeU1sc0tab2hraUNJejFtdnhKRlVNdGQ2Z1MzTHF0UFc4?=
 =?utf-8?B?eUJxMEdEMlV1MC9zV3h6RU9VZXZmMkdkcUh1VEdjMDRjWDZtUjNxemhqbXVx?=
 =?utf-8?B?WkZlWURKL0xHTVlTelpuQjJiWkIyRnpEUWZ2L0JvMGgwaFBXejNta1BMMXRm?=
 =?utf-8?B?RzJWWFNvdWZlSXpJOFZUdkhKZ3R6Y2RIWURETHlKUVdYNXB1d1NMZnJPeGxt?=
 =?utf-8?B?RUNaT1cvMXU1ekVRak5jQlQvRW11emJ1ZGdmcngyMXpNWUxDNndsdEp5YitS?=
 =?utf-8?B?bVV3OXpaQVU3akJMZGtmTFBHemVsWWNyZlFXVUhleDcxYm1JcTNsd0RnOHFZ?=
 =?utf-8?B?Tkx4ZjdFcWlxRlZFZUpPdDltUFRsRHVkM2grMEhYbC9tYlRtNTNBNDFWN09V?=
 =?utf-8?B?eGVHaW04Z3phSWEvQ2pJaWl0dHU1SktRMDdsbkJLbTUwMTVrUVduR09mcjhn?=
 =?utf-8?B?dlVyMFM3SnRab2h1QmoyWmVaWUxTaytoVm5kam9kcFJ1MmhHZWRXWjhYRzg2?=
 =?utf-8?Q?zWk6mYFqEHSHnIP20P8DxUp3e?=
X-MS-Exchange-CrossTenant-Network-Message-Id: c778b8e3-36d6-4968-8c4a-08dc6e2a296d
X-MS-Exchange-CrossTenant-AuthSource: BL1PR11MB5978.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 May 2024 00:10:54.4645
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ALFTYmwvWeCUDgXuff7j8UKuYWsIExsYqtygyW1Ws7yMEihqVi8XB3AU9fzZfpXZrDM8d7x+96fEkrcZTLDEOg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB5957
X-OriginatorOrg: intel.com


On 1/05/2024 7:51 am, Haitao Huang wrote:
>   
>   static void sgx_reclaim_pages_global(struct mm_struct *charge_mm)
>   {
> -	sgx_reclaim_pages(&sgx_global_lru, charge_mm);
> +	if (IS_ENABLED(CONFIG_CGROUP_MISC))
> +		sgx_cgroup_reclaim_pages(misc_cg_root(), charge_mm);
> +	else
> +		sgx_reclaim_pages(&sgx_global_lru, charge_mm);
>   }
>   

I think we have a problem here when we do global reclaim starting from 
the ROOT cgroup:

This function will mostly just only try to reclaim from the ROOT cgroup, 
but won't reclaim from the descendants.

The reason is the sgx_cgroup_reclaim_pages() will simply return after 
"scanning" SGX_NR_TO_SCAN (16) pages w/o going to the descendants, and 
the "scanning" here simply means "removing the EPC page from the 
cgroup's LRU list".

So as long as the ROOT cgroup LRU contains more than SGX_NR_TO_SCAN (16) 
pages, effectively sgx_cgroup_reclaim_pages() will just scan and return 
w/o going into the descendants.  Having 16 EPC pages should be a "almost 
always true" case I suppose.

When the sgx_reclaim_pages_global() is called again, we will start from 
the ROOT again.

That means the this doesn't truly reclaim "from global" at all.

IMHO the behaviour of sgx_cgroup_reclaim_pages() is OK for per-cgroup 
reclaim because I think in this case our intention is we should try best 
to reclaim from the cgroup, i.e., whether we can reclaim from 
descendants doesn't matter.

But for global reclaim this doesn't work.

Am I missing anything?