Received-SPF: pass (google.com: domain of linux-kernel+bounces-79003-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Content-Type: text/plain; charset=iso-8859-15; format=flowed; delsp=yes
To: "hpa@zytor.com" <hpa@zytor.com>, "tim.c.chen@linux.intel.com"
 <tim.c.chen@linux.intel.com>, "linux-sgx@vger.kernel.org"
 <linux-sgx@vger.kernel.org>, "x86@kernel.org" <x86@kernel.org>,
 "dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
 "jarkko@kernel.org" <jarkko@kernel.org>, "cgroups@vger.kernel.org"
 <cgroups@vger.kernel.org>, "linux-kernel@vger.kernel.org"
 <linux-kernel@vger.kernel.org>, "mkoutny@suse.com" <mkoutny@suse.com>,
 "tglx@linutronix.de" <tglx@linutronix.de>, "Mehta, Sohil"
 <sohil.mehta@intel.com>, "tj@kernel.org" <tj@kernel.org>, "mingo@redhat.com"
 <mingo@redhat.com>, "bp@alien8.de" <bp@alien8.de>, "Huang, Kai"
 <kai.huang@intel.com>
Cc: "mikko.ylinen@linux.intel.com" <mikko.ylinen@linux.intel.com>,
 "seanjc@google.com" <seanjc@google.com>, "anakrish@microsoft.com"
 <anakrish@microsoft.com>, "Zhang, Bo" <zhanb@microsoft.com>,
 "kristen@linux.intel.com" <kristen@linux.intel.com>, "yangjie@microsoft.com"
 <yangjie@microsoft.com>, "Li, Zhiquan1" <zhiquan1.li@intel.com>,
 "chrisyan@microsoft.com" <chrisyan@microsoft.com>
Subject: Re: [PATCH v9 13/15] x86/sgx: Turn on per-cgroup EPC reclamation
References: <20240205210638.157741-1-haitao.huang@linux.intel.com>
 <20240205210638.157741-14-haitao.huang@linux.intel.com>
 <87a85645ef1661e54ae6e56f1e47db25c3f8d7af.camel@intel.com>
 <op.2jjxqjiwwjvjmi@hhuan26-mobl.amr.corp.intel.com>
 <50ecd28c-4514-4ca9-8eb7-4cae24ba9d1d@intel.com>
Date: Fri, 23 Feb 2024 12:46:40 -0600
Precedence: bulk
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
From: "Haitao Huang" <haitao.huang@linux.intel.com>
Organization: Intel
Message-ID: <op.2jlyf2o4wjvjmi@hhuan26-mobl.amr.corp.intel.com>
In-Reply-To: <50ecd28c-4514-4ca9-8eb7-4cae24ba9d1d@intel.com>
User-Agent: Opera Mail/1.0 (Win32)

On Thu, 22 Feb 2024 16:44:45 -0600, Huang, Kai <kai.huang@intel.com> wrote:

>
>
> On 23/02/2024 5:36 am, Haitao Huang wrote:
>> On Wed, 21 Feb 2024 05:23:00 -0600, Huang, Kai <kai.huang@intel.com>  
>> wrote:
>>
>>> On Mon, 2024-02-05 at 13:06 -0800, Haitao Huang wrote:
>>>> From: Kristen Carlson Accardi <kristen@linux.intel.com>
>>>>
>>>> Previous patches have implemented all infrastructure needed for
>>>> per-cgroup EPC page tracking and reclaiming. But all reclaimable EPC
>>>> pages are still tracked in the global LRU as sgx_lru_list() returns  
>>>> hard
>>>> coded reference to the global LRU.
>>>>
>>>> Change sgx_lru_list() to return the LRU of the cgroup in which the  
>>>> given
>>>> EPC page is allocated.
>>>>
>>>> This makes all EPC pages tracked in per-cgroup LRUs and the global
>>>> reclaimer (ksgxd) will not be able to reclaim any pages from the  
>>>> global
>>>> LRU. However, in cases of over-committing, i.e., sum of cgroup limits
>>>> greater than the total capacity, cgroups may never reclaim but the  
>>>> total
>>>> usage can still be near the capacity. Therefore global reclamation is
>>>> still needed in those cases and it should reclaim from the root  
>>>> cgroup.
>>>>
>>>> Modify sgx_reclaim_pages_global(), to reclaim from the root EPC cgroup
>>>> when cgroup is enabled, otherwise from the global LRU.
>>>>
>>>> Similarly, modify sgx_can_reclaim(), to check emptiness of LRUs of all
>>>> cgroups when EPC cgroup is enabled, otherwise only check the global  
>>>> LRU.
>>>>
>>>> With these changes, the global reclamation and per-cgroup reclamation
>>>> both work properly with all pages tracked in per-cgroup LRUs.
>>>>
>>>> Co-developed-by: Sean Christopherson <sean.j.christopherson@intel.com>
>>>> Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
>>>> Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
>>>> Co-developed-by: Haitao Huang <haitao.huang@linux.intel.com>
>>>> Signed-off-by: Haitao Huang <haitao.huang@linux.intel.com>
>>>> ---
>>>> V7:
>>>> - Split this out from the big patch, #10 in V6. (Dave, Kai)
>>>> ---
>>>>  arch/x86/kernel/cpu/sgx/main.c | 16 +++++++++++++++-
>>>>  1 file changed, 15 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/arch/x86/kernel/cpu/sgx/main.c  
>>>> b/arch/x86/kernel/cpu/sgx/main.c
>>>> index 6b0c26cac621..d4265a390ba9 100644
>>>> --- a/arch/x86/kernel/cpu/sgx/main.c
>>>> +++ b/arch/x86/kernel/cpu/sgx/main.c
>>>> @@ -34,12 +34,23 @@ static struct sgx_epc_lru_list sgx_global_lru;
>>>>
>>>>  static inline struct sgx_epc_lru_list *sgx_lru_list(struct  
>>>> sgx_epc_page *epc_page)
>>>>  {
>>>> +#ifdef CONFIG_CGROUP_SGX_EPC
>>>> +    if (epc_page->epc_cg)
>>>> +        return &epc_page->epc_cg->lru;
>>>> +
>>>> +    /* This should not happen if kernel is configured correctly */
>>>> +    WARN_ON_ONCE(1);
>>>> +#endif
>>>>      return &sgx_global_lru;
>>>>  }
>>>
>>> How about when EPC cgroup is enabled, but one enclave doesn't belong  
>>> to any EPC
>>> cgroup?  Is it OK to track EPC pages for these enclaves to the root  
>>> EPC cgroup's
>>> LRU list together with other enclaves belongs to the root cgroup?
>>>
>>>
>>> This should be a valid case, right?
>>  There is no such case. Each page is in the root by default.
>>
>
> Is it guaranteed by the (misc) cgroup design/implementation?  If so  
> please add this information to the changelog and/or comments?  It helps  
> non-cgroup expert like me to understand.
>

Will do

Thanks
Haitao