Message-ID: <233a2fa4355b060361b9e9a6dc98b3175d1143fb.camel@kernel.org>
Subject: Re: [PATCH v13 2/2] x86/sgx: Add an attribute for the amount of SGX
 memory in a NUMA node
From:   Jarkko Sakkinen <jarkko@kernel.org>
To:     Dave Hansen <dave.hansen@intel.com>, Borislav Petkov <bp@alien8.de>
Cc:     Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>,
        Dave Hansen <dave.hansen@linux.intel.com>, x86@kernel.org,
        "H. Peter Anvin" <hpa@zytor.com>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        "Rafael J. Wysocki" <rafael@kernel.org>, reinette.chatre@intel.com,
        linux-kernel@vger.kernel.org, linux-sgx@vger.kernel.org
Date:   Sat, 11 Dec 2021 17:37:46 +0200
In-Reply-To: <32abd0a7-bc3c-f7a7-3330-8287ef05131c@intel.com>
References: <20211116162116.93081-1-jarkko@kernel.org>
         <20211116162116.93081-2-jarkko@kernel.org>
         <f25d95e6-e129-597b-5d93-d7264feae8b8@intel.com> <YbCEl9kqhTz2iOY2@zn.tnic>
         <32abd0a7-bc3c-f7a7-3330-8287ef05131c@intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.42.2 
MIME-Version: 1.0
Precedence: bulk

On Wed, 2021-12-08 at 11:38 -0800, Dave Hansen wrote:
> I reworked the changelog quite a bit, addressing some of Borislav's
> questions.  No code changes, though.
>=20
> The result is below.  I've retained Greg's ack.  I'll stick this in
> tip/x86/sgx if this looks OK to everyone.
>=20
> ---
>=20
> From: Jarkko Sakkinen <jarkko@kernel.org>
>=20
> =3D=3D Problem =3D=3D
>=20
> The amount of SGX memory on a system is determined by the BIOS and it
> varies wildly between systems.  It can be as small as dozens of MB's
> and as large as many GB's on servers.  Just like how applications need
> to know how much regular RAM is available, enclave builders need to
> know how much SGX memory an enclave can consume.
>=20
> =3D=3D Solution =3D=3D
>=20
> Introduce a new sysfs file:
>=20
> 	/sys/devices/system/node/nodeX/x86/sgx_total_bytes
>=20
> to enumerate the amount of SGX memory available in each NUMA node.
> This serves the same function for SGX as /proc/meminfo or
> /sys/devices/system/node/nodeX/meminfo does for normal RAM.
>=20
> 'sgx_total_bytes' is needed today to help drive the SGX selftests.
> SGX-specific swap code is exercised by creating overcommitted enclaves
> which are larger than the physical SGX memory on the system.  They
> currently use a CPUID-based approach which can diverge from the actual
> amount of SGX memory available.  'sgx_total_bytes' ensures that the
> selftests can work efficiently and do not attempt stupid things like
> creating a 100,000 MB enclave on a system with 128 MB of SGX memory.
>=20
> =3D=3D Implementation Details =3D=3D
>=20
> Introduce CONFIG_HAVE_ARCH_NODE_DEV_GROUP opt-in flag to expose an
> arch specific attribute group, and add an attribute for the amount of
> SGX memory in bytes to each NUMA node:
>=20
> =3D=3D ABI Design Discussion =3D=3D
>=20
> As opposed to the per-node ABI, a single, global ABI was considered.
> However, this would prevent enclaves from being able to size
> themselves so that they fit on a single NUMA node.  Essentially, a
> single value would rule out NUMA optimizations for enclaves.
>=20
> Create a new "x86/" directory inside each "nodeX/" sysfs directory.
> 'sgx_total_bytes' is expected to be the first of at least a few
> sgx-specific files to be placed in the new directory.  Just scanning
> /proc/meminfo, these are the no-brainers that we have for RAM, but we
> need for SGX:
>=20
> 	MemTotal:       xxxx kB // sgx_total_bytes (implemented here)
> 	MemFree:        yyyy kB // sgx_free_bytes
> 	SwapTotal:      zzzz kB // sgx_swapped_bytes
>=20
> So, at *least* three.  I think we will eventually end up needing
> something more along the lines of a dozen.  A new directory (as
> opposed to being in the nodeX/ "root") directory avoids cluttering the
> root with several "sgx_*" files.
>=20
> Place the new file in a new "nodeX/x86/" directory because SGX is
> highly x86-specific.  It is very unlikely that any other architecture
> (or even non-Intel x86 vendor) will ever implement SGX.  Using "sgx/"
> as opposed to "x86/" was also considered.  But, there is a real chance
> this can get used for other arch-specific purposes.
>=20
> [ dhansen: rewrite changelog ]
>=20
> Signed-off-by: Jarkko Sakkinen <jarkko@kernel.org>
> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
> Acked-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> Link: https://lkml.kernel.org/r/20211116162116.93081-2-jarkko@kernel.org
> ---
>  Documentation/ABI/stable/sysfs-devices-node |  6 ++++++
>  arch/Kconfig                                |  4 ++++
>  arch/x86/Kconfig                            |  1 +
>  arch/x86/kernel/cpu/sgx/main.c              | 20 ++++++++++++++++++++
>  arch/x86/kernel/cpu/sgx/sgx.h               |  1 +
>  drivers/base/node.c                         |  3 +++
>  include/linux/numa.h                        |  4 ++++
>  7 files changed, 39 insertions(+)

Thank you! Looks good to me.

/Jarkko