Received: by 2002:a05:7412:b995:b0:f9:9502:5bb8 with SMTP id it21csp7682298rdb; Thu, 4 Jan 2024 04:38:57 -0800 (PST) X-Google-Smtp-Source: AGHT+IH9cyRMfmQ2BxjNbtFbaucJzccgF6ByTkJpYop9SBtHVe5lwnmHeTWR7lY+wfdVbUg93kqH X-Received: by 2002:a17:907:9216:b0:a26:ea22:e62b with SMTP id ka22-20020a170907921600b00a26ea22e62bmr290265ejb.43.1704371937590; Thu, 04 Jan 2024 04:38:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704371937; cv=none; d=google.com; s=arc-20160816; b=yAyrmfofRyytSKirQyLfXP5s3+NC79OuaXgxbwsobGJaRf/H3PyuxBcpcyfKhN3ak8 0eHpzxATy6VM7kcnH8JVnX2bg7a2TeItdq/L0merQULepxbGs0DSefHZ5sKrup1/ev1n w9aJQ83yHNo4I627msGF7tVCSc6O/b3mQUmd+x8FXW03hxJZX8tvorhGhY1SQmHmRbDA P310OJtrjzzIi0vPNd1jc33WJ8lx6PVrUI4g2FpzJI8bGtXwW0QZqrTxrghow0xIAaGE JLSTz3QCozDAoeJ3BBw/XnEeLm3skl0Ndv+qavR5KxGwP/fsQg+0jPuq3039l0+x4ZOi vyXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:content-disposition:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:message-id:subject:cc :to:from:date:dkim-signature:dkim-signature; bh=4uvQkYHYu1QazRbslMypDkupBVwWahN8iFalOSYDpT4=; fh=3Q7pz6nILi7igWoEiIC+6rC1yX3O3m+FVwkrVnXd9gU=; b=gR3P1HSe2pOf1jt9+w96UCkCg17TM41u1U4r6aCq1q/mZdeMlf8GF4Q9gFcmEmrO9q cWIhcWOp7zyeKZetR9pnKGpFdMxUtyyUPuJV0ZrwLSjCd1DC0GXXQCugema9oLOu2yq3 brD3dyeaZBKGLEz/VsxGl1dn9F/VbZkS42NKQsNuRn1//htCyjM7mejs1ydwRhIKo+Jb TMxidUJwZ3urGR1QDk6MGTfdiRQR+QZpZjh8JBZGyuJSuXpkAUQYrjGkegBcjvWn/mlP FlxLoB+AlsBqOnp0cBVUDb8krGX37h5+TMsorMflw3byaRI9m51y/d8awYMSCBf4ZcdV Zq3w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=XhYFJs26; dkim=pass header.i=@suse.com header.s=susede1 header.b=XhYFJs26; spf=pass (google.com: domain of linux-kernel+bounces-16651-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-16651-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Return-Path: Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id l22-20020a1709067d5600b00a2355010ba6si11667335ejp.1033.2024.01.04.04.38.57 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Jan 2024 04:38:57 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-16651-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=XhYFJs26; dkim=pass header.i=@suse.com header.s=susede1 header.b=XhYFJs26; spf=pass (google.com: domain of linux-kernel+bounces-16651-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-16651-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 2FB4C1F252D6 for ; Thu, 4 Jan 2024 12:38:57 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2143C2137C; Thu, 4 Jan 2024 12:38:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b="XhYFJs26"; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b="XhYFJs26" X-Original-To: linux-kernel@vger.kernel.org Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 728642137E; Thu, 4 Jan 2024 12:38:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.com Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 43552220AB; Thu, 4 Jan 2024 12:38:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1704371923; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=4uvQkYHYu1QazRbslMypDkupBVwWahN8iFalOSYDpT4=; b=XhYFJs26rRAOJl7Zn6+UGJtoGHtlajogsGKFwi/XTOrabn1JfBZAgXhU+yRMGgS7cQnuA7 ay+cV55cOpbbrbcxbdn/WM8q/0X+lhPEl8qRbTTzK4T90OLSSx748Vy8x+tHsrGlwDypr7 LhFZ8uZjFlo4UaDJpcwNuMmnTfVtWwo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1704371923; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=4uvQkYHYu1QazRbslMypDkupBVwWahN8iFalOSYDpT4=; b=XhYFJs26rRAOJl7Zn6+UGJtoGHtlajogsGKFwi/XTOrabn1JfBZAgXhU+yRMGgS7cQnuA7 ay+cV55cOpbbrbcxbdn/WM8q/0X+lhPEl8qRbTTzK4T90OLSSx748Vy8x+tHsrGlwDypr7 LhFZ8uZjFlo4UaDJpcwNuMmnTfVtWwo= Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id CFD97137E8; Thu, 4 Jan 2024 12:38:42 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap1.dmz-prg2.suse.org with ESMTPSA id gmCwMdKmlmWaHQAAD6G6ig (envelope-from ); Thu, 04 Jan 2024 12:38:42 +0000 Date: Thu, 4 Jan 2024 13:38:41 +0100 From: Michal =?utf-8?Q?Koutn=C3=BD?= To: Haitao Huang Cc: "Mehta, Sohil" , "jarkko@kernel.org" , "x86@kernel.org" , "dave.hansen@linux.intel.com" , "cgroups@vger.kernel.org" , "hpa@zytor.com" , "mingo@redhat.com" , "tj@kernel.org" , "tglx@linutronix.de" , "linux-sgx@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "bp@alien8.de" , "Huang, Kai" , "mikko.ylinen@linux.intel.com" , "seanjc@google.com" , "Zhang, Bo" , "kristen@linux.intel.com" , "anakrish@microsoft.com" , "sean.j.christopherson@intel.com" , "Li, Zhiquan1" , "yangjie@microsoft.com" Subject: Re: [PATCH v6 09/12] x86/sgx: Restructure top-level EPC reclaim function Message-ID: References: <20231030182013.40086-1-haitao.huang@linux.intel.com> <20231030182013.40086-10-haitao.huang@linux.intel.com> <431c5d7f5aee7d11ec2e8aa2e526fde438fa53b4.camel@intel.com> <3c27bca678c1b041920a14a7da0d958c9861ebca.camel@intel.com> <73ed579be8ad81835df1c309b7c69b491b7f2c8e.camel@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="6xoj57vgctb6e4ry" Content-Disposition: inline In-Reply-To: X-Spam-Level: X-Rspamd-Server: rspamd1.dmz-prg2.suse.org X-Spam-Level: X-Spam-Flag: NO X-Spamd-Result: default: False [-1.91 / 50.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[suse.com:s=susede1]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; RCVD_COUNT_THREE(0.00)[3]; DKIM_SIGNED(0.00)[suse.com:s=susede1]; DKIM_TRACE(0.00)[suse.com:+]; MX_GOOD(-0.01)[]; RCPT_COUNT_TWELVE(0.00)[22]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.com:dkim,intel.com:email]; SIGNED_PGP(-2.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; MID_RHS_NOT_FQDN(0.50)[]; RCVD_TLS_ALL(0.00)[]; BAYES_HAM(-0.00)[10.53%]; RBL_SPAMHAUS_BLOCKED_OPENRESOLVER(0.00)[2a07:de40:b281:104:10:150:64:97:from] Authentication-Results: smtp-out1.suse.de; dkim=pass header.d=suse.com header.s=susede1 header.b=XhYFJs26 X-Spam-Score: -1.91 X-Rspamd-Queue-Id: 43552220AB --6xoj57vgctb6e4ry Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hello. On Mon, Dec 18, 2023 at 03:24:40PM -0600, Haitao Huang wrote: > Thanks for raising this. Actually my understanding the above discussion w= as > mainly about not doing reclaiming by killing enclaves, i.e., I assumed > "reclaiming" within that context only meant for that particular kind. >=20 > As Mikko pointed out, without reclaiming per-cgroup, the max limit of each > cgroup needs to accommodate the peak usage of enclaves within that cgroup. > That may be inconvenient for practical usage and limits could be forced to > be set larger than necessary to run enclaves performantly. For example, we > can observe following undesired consequences comparing a system with curr= ent > kernel loaded with enclaves whose total peak usage is greater than the EPC > capacity. >=20 > 1) If a user wants to load the same exact enclaves but in separate cgroup= s, > then the sum of cgroup limits must be higher than the capacity and the > system will end up doing the same old global reclaiming as it is currently > doing. Cgroup is not useful at all for isolating EPC consumptions. That is the use of limits to prevent a runaway cgroup smothering the system. Overcommited values in such a config are fine because the more simultaneous runaways, the less likely. The peak consumption is on the fair expense of others (some efficiency) and the limit contains the runaway (hence the isolation). > 2) To isolate impact of usage of each cgroup on other cgroups and yet sti= ll > being able to load each enclave, the user basically has to carefully plan= to > ensure the sum of cgroup max limits, i.e., the sum of peak usage of > enclaves, is not reaching over the capacity. That means no over-commiting > allowed and the same system may not be able to load as many enclaves as w= ith > current kernel. Another "config layout" of limits is to achieve partitioning (sum =3D=3D capacity). That is perfect isolation but it naturally goes against efficient utilization. The way other controllers approach this trade-off is with weights (cpu, io) or protections (memory). I'm afraid misc controller is not ready for this. My opinion is to start with the simple limits (first patches) and think of prioritization/guarantee mechanism based on real cases. HTH, Michal --6xoj57vgctb6e4ry Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iHUEABYKAB0WIQQpEWyjXuwGT2dDBqAGvrMr/1gcjgUCZZamzwAKCRAGvrMr/1gc jh5HAQC7JKavOsXc5+CL5ZJfrJhI3she9MQsyt/LmjPqO9lFcwEAq8+16AXaKO23 JWl2/b7BSmoBdumcmAz8DkzUXD7GIg0= =kG33 -----END PGP SIGNATURE----- --6xoj57vgctb6e4ry--