Received-SPF: pass (google.com: domain of linux-kernel+bounces-16651-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Date: Thu, 4 Jan 2024 13:38:41 +0100
From: Michal =?utf-8?Q?Koutn=C3=BD?= <mkoutny@suse.com>
To: Haitao Huang <haitao.huang@linux.intel.com>
Cc: "Mehta, Sohil" <sohil.mehta@intel.com>, 
	"jarkko@kernel.org" <jarkko@kernel.org>, "x86@kernel.org" <x86@kernel.org>, 
	"dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>, "cgroups@vger.kernel.org" <cgroups@vger.kernel.org>, 
	"hpa@zytor.com" <hpa@zytor.com>, "mingo@redhat.com" <mingo@redhat.com>, 
	"tj@kernel.org" <tj@kernel.org>, "tglx@linutronix.de" <tglx@linutronix.de>, 
	"linux-sgx@vger.kernel.org" <linux-sgx@vger.kernel.org>, "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, 
	"bp@alien8.de" <bp@alien8.de>, "Huang, Kai" <kai.huang@intel.com>, 
	"mikko.ylinen@linux.intel.com" <mikko.ylinen@linux.intel.com>, "seanjc@google.com" <seanjc@google.com>, 
	"Zhang, Bo" <zhanb@microsoft.com>, "kristen@linux.intel.com" <kristen@linux.intel.com>, 
	"anakrish@microsoft.com" <anakrish@microsoft.com>, 
	"sean.j.christopherson@intel.com" <sean.j.christopherson@intel.com>, "Li, Zhiquan1" <zhiquan1.li@intel.com>, 
	"yangjie@microsoft.com" <yangjie@microsoft.com>
Subject: Re: [PATCH v6 09/12] x86/sgx: Restructure top-level EPC reclaim
 function
Message-ID: <mmcdj5ulscoxyofetdgmygytd2dnjusnoj3ypix2plnwp7nbks@bqesmrpmsca7>
References: <20231030182013.40086-1-haitao.huang@linux.intel.com>
 <20231030182013.40086-10-haitao.huang@linux.intel.com>
 <c8fc40dc56b853fbff14ba22db197c80a6d31820.camel@intel.com>
 <op.2e0yod2lwjvjmi@hhuan26-mobl.amr.corp.intel.com>
 <431c5d7f5aee7d11ec2e8aa2e526fde438fa53b4.camel@intel.com>
 <op.2ftmyampwjvjmi@hhuan26-mobl.amr.corp.intel.com>
 <3c27bca678c1b041920a14a7da0d958c9861ebca.camel@intel.com>
 <op.2f0eo8r1wjvjmi@hhuan26-mobl.amr.corp.intel.com>
 <73ed579be8ad81835df1c309b7c69b491b7f2c8e.camel@intel.com>
 <op.2f523elowjvjmi@hhuan26-mobl.amr.corp.intel.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="6xoj57vgctb6e4ry"
Content-Disposition: inline
In-Reply-To: <op.2f523elowjvjmi@hhuan26-mobl.amr.corp.intel.com>


--6xoj57vgctb6e4ry
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Hello.

On Mon, Dec 18, 2023 at 03:24:40PM -0600, Haitao Huang <haitao.huang@linux.=
intel.com> wrote:
> Thanks for raising this. Actually my understanding the above discussion w=
as
> mainly about not doing reclaiming by killing enclaves, i.e., I assumed
> "reclaiming" within that context only meant for that particular kind.
>=20
> As Mikko pointed out, without reclaiming per-cgroup, the max limit of each
> cgroup needs to accommodate the peak usage of enclaves within that cgroup.
> That may be inconvenient for practical usage and limits could be forced to
> be set larger than necessary to run enclaves performantly. For example, we
> can observe following undesired consequences comparing a system with curr=
ent
> kernel loaded with enclaves whose total peak usage is greater than the EPC
> capacity.
>=20
> 1) If a user wants to load the same exact enclaves but in separate cgroup=
s,
> then the sum of cgroup limits must be higher than the capacity and the
> system will end up doing the same old global reclaiming as it is currently
> doing. Cgroup is not useful at all for isolating EPC consumptions.

That is the use of limits to prevent a runaway cgroup smothering the
system. Overcommited values in such a config are fine because the more
simultaneous runaways, the less likely.
The peak consumption is on the fair expense of others (some efficiency)
and the limit contains the runaway (hence the isolation).

> 2) To isolate impact of usage of each cgroup on other cgroups and yet sti=
ll
> being able to load each enclave, the user basically has to carefully plan=
 to
> ensure the sum of cgroup max limits, i.e., the sum of peak usage of
> enclaves, is not reaching over the capacity. That means no over-commiting
> allowed and the same system may not be able to load as many enclaves as w=
ith
> current kernel.

Another "config layout" of limits is to achieve partitioning (sum =3D=3D
capacity). That is perfect isolation but it naturally goes against
efficient utilization. The way other controllers approach this trade-off
is with weights (cpu, io) or protections (memory). I'm afraid misc
controller is not ready for this.

My opinion is to start with the simple limits (first patches) and think
of prioritization/guarantee mechanism based on real cases.

HTH,
Michal

--6xoj57vgctb6e4ry
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iHUEABYKAB0WIQQpEWyjXuwGT2dDBqAGvrMr/1gcjgUCZZamzwAKCRAGvrMr/1gc
jh5HAQC7JKavOsXc5+CL5ZJfrJhI3she9MQsyt/LmjPqO9lFcwEAq8+16AXaKO23
JWl2/b7BSmoBdumcmAz8DkzUXD7GIg0=
=kG33
-----END PGP SIGNATURE-----

--6xoj57vgctb6e4ry--