Received-SPF: pass (google.com: domain of linux-kernel+bounces-17149-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Content-Type: text/plain; charset=iso-8859-15; format=flowed; delsp=yes
To: =?iso-8859-15?Q?Michal_Koutn=FD?= <mkoutny@suse.com>
Cc: "Mehta, Sohil" <sohil.mehta@intel.com>, "jarkko@kernel.org"
 <jarkko@kernel.org>, "x86@kernel.org" <x86@kernel.org>,
 "dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
 "cgroups@vger.kernel.org" <cgroups@vger.kernel.org>, "hpa@zytor.com"
 <hpa@zytor.com>, "mingo@redhat.com" <mingo@redhat.com>, "tj@kernel.org"
 <tj@kernel.org>, "tglx@linutronix.de" <tglx@linutronix.de>,
 "linux-sgx@vger.kernel.org" <linux-sgx@vger.kernel.org>,
 "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, "bp@alien8.de"
 <bp@alien8.de>, "Huang, Kai" <kai.huang@intel.com>,
 "mikko.ylinen@linux.intel.com" <mikko.ylinen@linux.intel.com>,
 "seanjc@google.com" <seanjc@google.com>, "Zhang, Bo" <zhanb@microsoft.com>,
 "kristen@linux.intel.com" <kristen@linux.intel.com>, "anakrish@microsoft.com"
 <anakrish@microsoft.com>, "sean.j.christopherson@intel.com"
 <sean.j.christopherson@intel.com>, "Li, Zhiquan1" <zhiquan1.li@intel.com>,
 "yangjie@microsoft.com" <yangjie@microsoft.com>
Subject: Re: [PATCH v6 09/12] x86/sgx: Restructure top-level EPC reclaim
 function
References: <20231030182013.40086-1-haitao.huang@linux.intel.com>
 <20231030182013.40086-10-haitao.huang@linux.intel.com>
 <c8fc40dc56b853fbff14ba22db197c80a6d31820.camel@intel.com>
 <op.2e0yod2lwjvjmi@hhuan26-mobl.amr.corp.intel.com>
 <431c5d7f5aee7d11ec2e8aa2e526fde438fa53b4.camel@intel.com>
 <op.2ftmyampwjvjmi@hhuan26-mobl.amr.corp.intel.com>
 <3c27bca678c1b041920a14a7da0d958c9861ebca.camel@intel.com>
 <op.2f0eo8r1wjvjmi@hhuan26-mobl.amr.corp.intel.com>
 <73ed579be8ad81835df1c309b7c69b491b7f2c8e.camel@intel.com>
 <op.2f523elowjvjmi@hhuan26-mobl.amr.corp.intel.com>
 <mmcdj5ulscoxyofetdgmygytd2dnjusnoj3ypix2plnwp7nbks@bqesmrpmsca7>
Date: Thu, 04 Jan 2024 13:20:38 -0600
Precedence: bulk
MIME-Version: 1.0
Content-Transfer-Encoding: Quoted-Printable
From: "Haitao Huang" <haitao.huang@linux.intel.com>
Organization: Intel
Message-ID: <op.2g1eoojxwjvjmi@hhuan26-mobl.amr.corp.intel.com>
In-Reply-To: <mmcdj5ulscoxyofetdgmygytd2dnjusnoj3ypix2plnwp7nbks@bqesmrpmsca7>
User-Agent: Opera Mail/1.0 (Win32)

Hi Michal,

On Thu, 04 Jan 2024 06:38:41 -0600, Michal Koutn=FD <mkoutny@suse.com> w=
rote:

> Hello.
>
> On Mon, Dec 18, 2023 at 03:24:40PM -0600, Haitao Huang  =

> <haitao.huang@linux.intel.com> wrote:
>> Thanks for raising this. Actually my understanding the above discussi=
on  =

>> was
>> mainly about not doing reclaiming by killing enclaves, i.e., I assume=
d
>> "reclaiming" within that context only meant for that particular kind.=

>>
>> As Mikko pointed out, without reclaiming per-cgroup, the max limit of=
  =

>> each
>> cgroup needs to accommodate the peak usage of enclaves within that  =

>> cgroup.
>> That may be inconvenient for practical usage and limits could be forc=
ed  =

>> to
>> be set larger than necessary to run enclaves performantly. For exampl=
e,  =

>> we
>> can observe following undesired consequences comparing a system with =
 =

>> current
>> kernel loaded with enclaves whose total peak usage is greater than th=
e  =

>> EPC
>> capacity.
>>
>> 1) If a user wants to load the same exact enclaves but in separate  =

>> cgroups,
>> then the sum of cgroup limits must be higher than the capacity and th=
e
>> system will end up doing the same old global reclaiming as it is  =

>> currently
>> doing. Cgroup is not useful at all for isolating EPC consumptions.
>
> That is the use of limits to prevent a runaway cgroup smothering the
> system. Overcommited values in such a config are fine because the more=

> simultaneous runaways, the less likely.
> The peak consumption is on the fair expense of others (some efficiency=
)
> and the limit contains the runaway (hence the isolation).
>

This makes sense to me in theory. Mikko, Chris Y/Bo Z, your thoughts on =
 =

whether this is good enough for your intended usages?

>> 2) To isolate impact of usage of each cgroup on other cgroups and yet=
  =

>> still
>> being able to load each enclave, the user basically has to carefully =
 =

>> plan to
>> ensure the sum of cgroup max limits, i.e., the sum of peak usage of
>> enclaves, is not reaching over the capacity. That means no  =

>> over-commiting
>> allowed and the same system may not be able to load as many enclaves =
as  =

>> with
>> current kernel.
>
> Another "config layout" of limits is to achieve partitioning (sum =3D=3D=

> capacity). That is perfect isolation but it naturally goes against
> efficient utilization. The way other controllers approach this trade-o=
ff
> is with weights (cpu, io) or protections (memory). I'm afraid misc
> controller is not ready for this.
>
> My opinion is to start with the simple limits (first patches) and thin=
k
> of prioritization/guarantee mechanism based on real cases.
>

We moved away from using mem like custom controller with (low, high, max=
)  =

to misc controller. But if we need add those down the road, then the  =

interface needs be changed. So my concern on this route would be whether=
  =

misc would allow any of those extensions. On the other hand, it might tu=
rn  =

out less complex just doing the reclamation per cgroup.

Thanks a lot for your comments and they are really helpful!

Haitao