The amount of SGX memory on the system is determined by the BIOS and it
varies wildly between systems. It can be from dozens of MB's on desktops
or VM's, up to many GB's on servers. Just like for regular memory, it is
sometimes useful to know the amount of usable SGX memory in the system.
Add SGX_MemTotal field to /proc/meminfo, which shows the total amount of
usable SGX memory in the system. E.g. with 32 MB reserved for SGX from
BIOS, the printout would be:
SGX_MemTotal: 22528 kB
It is less than 32 MB because some of the space is reserved for Enclave
Page Cache Metadata (EPCM), which contains state variables for all the
pages in the Enclave Page Cache (EPC). The latter contains the pages,
which applications can use to create enclaves.
Signed-off-by: Jarkko Sakkinen <[email protected]>
---
Documentation/x86/sgx.rst | 6 ++++++
arch/x86/include/asm/sgx.h | 10 +++++++---
arch/x86/kernel/cpu/sgx/main.c | 7 ++++++-
arch/x86/mm/pat/set_memory.c | 5 +++++
4 files changed, 24 insertions(+), 4 deletions(-)
diff --git a/Documentation/x86/sgx.rst b/Documentation/x86/sgx.rst
index dd0ac96ff9ef..68ee171e1d8f 100644
--- a/Documentation/x86/sgx.rst
+++ b/Documentation/x86/sgx.rst
@@ -250,3 +250,9 @@ user wants to deploy SGX applications both on the host and in guests
on the same machine, the user should reserve enough EPC (by taking out
total virtual EPC size of all SGX VMs from the physical EPC size) for
host SGX applications so they can run with acceptable performance.
+
+Supplemental fields for /proc/meminfo
+=====================================
+
+SGX_MemTotal
+ The total usable SGX protected memory in kilobytes.
diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
index 05f3e21f01a7..2ae9dc8c9411 100644
--- a/arch/x86/include/asm/sgx.h
+++ b/arch/x86/include/asm/sgx.h
@@ -365,6 +365,13 @@ struct sgx_sigstruct {
* comment!
*/
+#if defined(CONFIG_X86_SGX) || defined(CONFIG_X86_SGX_KVM)
+extern unsigned long sgx_nr_all_pages;
+
+int sgx_set_attribute(unsigned long *allowed_attributes,
+ unsigned int attribute_fd);
+#endif
+
#ifdef CONFIG_X86_SGX_KVM
int sgx_virt_ecreate(struct sgx_pageinfo *pageinfo, void __user *secs,
int *trapnr);
@@ -372,7 +379,4 @@ int sgx_virt_einit(void __user *sigstruct, void __user *token,
void __user *secs, u64 *lepubkeyhash, int *trapnr);
#endif
-int sgx_set_attribute(unsigned long *allowed_attributes,
- unsigned int attribute_fd);
-
#endif /* _ASM_X86_SGX_H */
diff --git a/arch/x86/kernel/cpu/sgx/main.c b/arch/x86/kernel/cpu/sgx/main.c
index 63d3de02bbcc..1fe26a8e80dc 100644
--- a/arch/x86/kernel/cpu/sgx/main.c
+++ b/arch/x86/kernel/cpu/sgx/main.c
@@ -28,7 +28,10 @@ static DECLARE_WAIT_QUEUE_HEAD(ksgxd_waitq);
static LIST_HEAD(sgx_active_page_list);
static DEFINE_SPINLOCK(sgx_reclaimer_lock);
-/* The free page list lock protected variables prepend the lock. */
+/* The number of usable EPC pages in the system. */
+unsigned long sgx_nr_all_pages;
+
+/* The number of free EPC pages in all nodes. */
static unsigned long sgx_nr_free_pages;
/* Nodes with one or more EPC sections. */
@@ -656,6 +659,8 @@ static bool __init sgx_setup_epc_section(u64 phys_addr, u64 size,
list_add_tail(§ion->pages[i].list, &sgx_dirty_page_list);
}
+ sgx_nr_all_pages += nr_pages;
+
return true;
}
diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c
index ad8a5c586a35..82bb09c298de 100644
--- a/arch/x86/mm/pat/set_memory.c
+++ b/arch/x86/mm/pat/set_memory.c
@@ -29,6 +29,7 @@
#include <asm/proto.h>
#include <asm/memtype.h>
#include <asm/set_memory.h>
+#include <asm/sgx.h>
#include "../mm_internal.h"
@@ -116,6 +117,10 @@ void arch_report_meminfo(struct seq_file *m)
if (direct_gbpages)
seq_printf(m, "DirectMap1G: %8lu kB\n",
direct_pages_count[PG_LEVEL_1G] << 20);
+
+#if defined(CONFIG_X86_SGX) || defined(CONFIG_X86_SGX_KVM)
+ seq_printf(m, "SGX_MemTotal: %8lu kB\n", sgx_nr_all_pages << 2);
+#endif
}
#else
static inline void split_page_count(int level) { }
--
2.25.1
On 2021-08-18 15:25, Jarkko Sakkinen wrote:
> The amount of SGX memory on the system is determined by the BIOS and it
> varies wildly between systems. It can be from dozens of MB's on desktops
> or VM's, up to many GB's on servers. Just like for regular memory, it is
> sometimes useful to know the amount of usable SGX memory in the system.
>
> Add SGX_MemTotal field to /proc/meminfo, which shows the total amount of
> usable SGX memory in the system. E.g. with 32 MB reserved for SGX from
> BIOS, the printout would be:
>
> SGX_MemTotal: 22528 kB
>
> It is less than 32 MB because some of the space is reserved for Enclave
> Page Cache Metadata (EPCM), which contains state variables for all the
> pages in the Enclave Page Cache (EPC). The latter contains the pages,
> which applications can use to create enclaves.
>
> Signed-off-by: Jarkko Sakkinen <[email protected]>
> ---
> Documentation/x86/sgx.rst | 6 ++++++
> arch/x86/include/asm/sgx.h | 10 +++++++---
> arch/x86/kernel/cpu/sgx/main.c | 7 ++++++-
> arch/x86/mm/pat/set_memory.c | 5 +++++
> 4 files changed, 24 insertions(+), 4 deletions(-)
>
> diff --git a/Documentation/x86/sgx.rst b/Documentation/x86/sgx.rst
> index dd0ac96ff9ef..68ee171e1d8f 100644
> --- a/Documentation/x86/sgx.rst
> +++ b/Documentation/x86/sgx.rst
> @@ -250,3 +250,9 @@ user wants to deploy SGX applications both on the host and in guests
> on the same machine, the user should reserve enough EPC (by taking out
> total virtual EPC size of all SGX VMs from the physical EPC size) for
> host SGX applications so they can run with acceptable performance.
> +
> +Supplemental fields for /proc/meminfo
> +=====================================
> +
> +SGX_MemTotal
> + The total usable SGX protected memory in kilobytes.
> diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
> index 05f3e21f01a7..2ae9dc8c9411 100644
> --- a/arch/x86/include/asm/sgx.h
> +++ b/arch/x86/include/asm/sgx.h
> @@ -365,6 +365,13 @@ struct sgx_sigstruct {
> * comment!
> */
>
> +#if defined(CONFIG_X86_SGX) || defined(CONFIG_X86_SGX_KVM)
> +extern unsigned long sgx_nr_all_pages;
> +
> +int sgx_set_attribute(unsigned long *allowed_attributes,
> + unsigned int attribute_fd);
> +#endif
> +
> #ifdef CONFIG_X86_SGX_KVM
> int sgx_virt_ecreate(struct sgx_pageinfo *pageinfo, void __user *secs,
> int *trapnr);
> @@ -372,7 +379,4 @@ int sgx_virt_einit(void __user *sigstruct, void __user *token,
> void __user *secs, u64 *lepubkeyhash, int *trapnr);
> #endif
>
> -int sgx_set_attribute(unsigned long *allowed_attributes,
> - unsigned int attribute_fd);
> -
This change seems unrelated?
> #endif /* _ASM_X86_SGX_H */
> diff --git a/arch/x86/kernel/cpu/sgx/main.c b/arch/x86/kernel/cpu/sgx/main.c
> index 63d3de02bbcc..1fe26a8e80dc 100644
> --- a/arch/x86/kernel/cpu/sgx/main.c
> +++ b/arch/x86/kernel/cpu/sgx/main.c
> @@ -28,7 +28,10 @@ static DECLARE_WAIT_QUEUE_HEAD(ksgxd_waitq);
> static LIST_HEAD(sgx_active_page_list);
> static DEFINE_SPINLOCK(sgx_reclaimer_lock);
>
> -/* The free page list lock protected variables prepend the lock. */
> +/* The number of usable EPC pages in the system. */
> +unsigned long sgx_nr_all_pages;
> +
> +/* The number of free EPC pages in all nodes. */
> static unsigned long sgx_nr_free_pages;
>
> /* Nodes with one or more EPC sections. */
> @@ -656,6 +659,8 @@ static bool __init sgx_setup_epc_section(u64 phys_addr, u64 size,
> list_add_tail(§ion->pages[i].list, &sgx_dirty_page_list);
> }
>
> + sgx_nr_all_pages += nr_pages;
> +
> return true;
> }
>
> diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c
> index ad8a5c586a35..82bb09c298de 100644
> --- a/arch/x86/mm/pat/set_memory.c
> +++ b/arch/x86/mm/pat/set_memory.c
> @@ -29,6 +29,7 @@
> #include <asm/proto.h>
> #include <asm/memtype.h>
> #include <asm/set_memory.h>
> +#include <asm/sgx.h>
>
> #include "../mm_internal.h"
>
> @@ -116,6 +117,10 @@ void arch_report_meminfo(struct seq_file *m)
> if (direct_gbpages)
> seq_printf(m, "DirectMap1G: %8lu kB\n",
> direct_pages_count[PG_LEVEL_1G] << 20);
> +
> +#if defined(CONFIG_X86_SGX) || defined(CONFIG_X86_SGX_KVM)
> + seq_printf(m, "SGX_MemTotal: %8lu kB\n", sgx_nr_all_pages << 2);
> +#endif
> }
> #else
> static inline void split_page_count(int level) { }
>
On Wed, Aug 18, 2021 at 03:29:59PM +0200, Jethro Beekman wrote:
> On 2021-08-18 15:25, Jarkko Sakkinen wrote:
> > The amount of SGX memory on the system is determined by the BIOS and it
> > varies wildly between systems. It can be from dozens of MB's on desktops
> > or VM's, up to many GB's on servers. Just like for regular memory, it is
> > sometimes useful to know the amount of usable SGX memory in the system.
> >
> > Add SGX_MemTotal field to /proc/meminfo, which shows the total amount of
> > usable SGX memory in the system. E.g. with 32 MB reserved for SGX from
> > BIOS, the printout would be:
> >
> > SGX_MemTotal: 22528 kB
> >
> > It is less than 32 MB because some of the space is reserved for Enclave
> > Page Cache Metadata (EPCM), which contains state variables for all the
> > pages in the Enclave Page Cache (EPC). The latter contains the pages,
> > which applications can use to create enclaves.
> >
> > Signed-off-by: Jarkko Sakkinen <[email protected]>
> > ---
> > Documentation/x86/sgx.rst | 6 ++++++
> > arch/x86/include/asm/sgx.h | 10 +++++++---
> > arch/x86/kernel/cpu/sgx/main.c | 7 ++++++-
> > arch/x86/mm/pat/set_memory.c | 5 +++++
> > 4 files changed, 24 insertions(+), 4 deletions(-)
> >
> > diff --git a/Documentation/x86/sgx.rst b/Documentation/x86/sgx.rst
> > index dd0ac96ff9ef..68ee171e1d8f 100644
> > --- a/Documentation/x86/sgx.rst
> > +++ b/Documentation/x86/sgx.rst
> > @@ -250,3 +250,9 @@ user wants to deploy SGX applications both on the host and in guests
> > on the same machine, the user should reserve enough EPC (by taking out
> > total virtual EPC size of all SGX VMs from the physical EPC size) for
> > host SGX applications so they can run with acceptable performance.
> > +
> > +Supplemental fields for /proc/meminfo
> > +=====================================
> > +
> > +SGX_MemTotal
> > + The total usable SGX protected memory in kilobytes.
> > diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
> > index 05f3e21f01a7..2ae9dc8c9411 100644
> > --- a/arch/x86/include/asm/sgx.h
> > +++ b/arch/x86/include/asm/sgx.h
> > @@ -365,6 +365,13 @@ struct sgx_sigstruct {
> > * comment!
> > */
> >
> > +#if defined(CONFIG_X86_SGX) || defined(CONFIG_X86_SGX_KVM)
> > +extern unsigned long sgx_nr_all_pages;
> > +
> > +int sgx_set_attribute(unsigned long *allowed_attributes,
> > + unsigned int attribute_fd);
> > +#endif
> > +
> > #ifdef CONFIG_X86_SGX_KVM
> > int sgx_virt_ecreate(struct sgx_pageinfo *pageinfo, void __user *secs,
> > int *trapnr);
> > @@ -372,7 +379,4 @@ int sgx_virt_einit(void __user *sigstruct, void __user *token,
> > void __user *secs, u64 *lepubkeyhash, int *trapnr);
> > #endif
> >
> > -int sgx_set_attribute(unsigned long *allowed_attributes,
> > - unsigned int attribute_fd);
> > -
>
> This change seems unrelated?
It's just a good practice not to define symbols that do not exist, so that
if the symbol is ever used, we get a compilation error, not linking error.
Since this is included to set_memory.c, based on this conclusion, I added
the check.
/Jarkko
On 8/18/21 7:40 AM, Jarkko Sakkinen wrote:
> On Wed, Aug 18, 2021 at 03:29:59PM +0200, Jethro Beekman wrote:
>> On 2021-08-18 15:25, Jarkko Sakkinen wrote:
>>> The amount of SGX memory on the system is determined by the BIOS and it
>>> varies wildly between systems. It can be from dozens of MB's on desktops
>>> or VM's, up to many GB's on servers. Just like for regular memory, it is
>>> sometimes useful to know the amount of usable SGX memory in the system.
>>>
>>> Add SGX_MemTotal field to /proc/meminfo, which shows the total amount of
>>> usable SGX memory in the system. E.g. with 32 MB reserved for SGX from
>>> BIOS, the printout would be:
>>>
>>> SGX_MemTotal: 22528 kB
>>>
>>> It is less than 32 MB because some of the space is reserved for Enclave
>>> Page Cache Metadata (EPCM), which contains state variables for all the
>>> pages in the Enclave Page Cache (EPC). The latter contains the pages,
>>> which applications can use to create enclaves.
>>>
>>> Signed-off-by: Jarkko Sakkinen <[email protected]>
>>> ---
>>> Documentation/x86/sgx.rst | 6 ++++++
>>> arch/x86/include/asm/sgx.h | 10 +++++++---
>>> arch/x86/kernel/cpu/sgx/main.c | 7 ++++++-
>>> arch/x86/mm/pat/set_memory.c | 5 +++++
>>> 4 files changed, 24 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/Documentation/x86/sgx.rst b/Documentation/x86/sgx.rst
>>> index dd0ac96ff9ef..68ee171e1d8f 100644
>>> --- a/Documentation/x86/sgx.rst
>>> +++ b/Documentation/x86/sgx.rst
>>> @@ -250,3 +250,9 @@ user wants to deploy SGX applications both on the host and in guests
>>> on the same machine, the user should reserve enough EPC (by taking out
>>> total virtual EPC size of all SGX VMs from the physical EPC size) for
>>> host SGX applications so they can run with acceptable performance.
>>> +
>>> +Supplemental fields for /proc/meminfo
>>> +=====================================
>>> +
>>> +SGX_MemTotal
>>> + The total usable SGX protected memory in kilobytes.
>>> diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
>>> index 05f3e21f01a7..2ae9dc8c9411 100644
>>> --- a/arch/x86/include/asm/sgx.h
>>> +++ b/arch/x86/include/asm/sgx.h
>>> @@ -365,6 +365,13 @@ struct sgx_sigstruct {
>>> * comment!
>>> */
>>>
>>> +#if defined(CONFIG_X86_SGX) || defined(CONFIG_X86_SGX_KVM)
>>> +extern unsigned long sgx_nr_all_pages;
>>> +
>>> +int sgx_set_attribute(unsigned long *allowed_attributes,
>>> + unsigned int attribute_fd);
>>> +#endif
>>> +
>>> #ifdef CONFIG_X86_SGX_KVM
>>> int sgx_virt_ecreate(struct sgx_pageinfo *pageinfo, void __user *secs,
>>> int *trapnr);
>>> @@ -372,7 +379,4 @@ int sgx_virt_einit(void __user *sigstruct, void __user *token,
>>> void __user *secs, u64 *lepubkeyhash, int *trapnr);
>>> #endif
>>>
>>> -int sgx_set_attribute(unsigned long *allowed_attributes,
>>> - unsigned int attribute_fd);
>>> -
>>
>> This change seems unrelated?
>
> It's just a good practice not to define symbols that do not exist, so that
> if the symbol is ever used, we get a compilation error, not linking error.
>
> Since this is included to set_memory.c, based on this conclusion, I added
> the check.
>
It would make sense to make this change in a separate patch since.
thanks,
-- Shuah
On Wed, 2021-08-18 at 12:34 -0600, Shuah Khan wrote:
> On 8/18/21 7:40 AM, Jarkko Sakkinen wrote:
> > On Wed, Aug 18, 2021 at 03:29:59PM +0200, Jethro Beekman wrote:
> > > On 2021-08-18 15:25, Jarkko Sakkinen wrote:
> > > > The amount of SGX memory on the system is determined by the
> > > > BIOS and it
> > > > varies wildly between systems. It can be from dozens of MB's
> > > > on desktops
> > > > or VM's, up to many GB's on servers. Just like for regular
> > > > memory, it is
> > > > sometimes useful to know the amount of usable SGX memory in the
> > > > system.
> > > >
> > > > Add SGX_MemTotal field to /proc/meminfo, which shows the total
> > > > amount of
> > > > usable SGX memory in the system. E.g. with 32 MB reserved for
> > > > SGX from
> > > > BIOS, the printout would be:
> > > >
> > > > SGX_MemTotal: 22528 kB
> > > >
> > > > It is less than 32 MB because some of the space is reserved for
> > > > Enclave
> > > > Page Cache Metadata (EPCM), which contains state variables for
> > > > all the
> > > > pages in the Enclave Page Cache (EPC). The latter contains the
> > > > pages,
> > > > which applications can use to create enclaves.
> > > >
> > > > Signed-off-by: Jarkko Sakkinen <[email protected]>
> > > > ---
> > > > Documentation/x86/sgx.rst | 6 ++++++
> > > > arch/x86/include/asm/sgx.h | 10 +++++++---
> > > > arch/x86/kernel/cpu/sgx/main.c | 7 ++++++-
> > > > arch/x86/mm/pat/set_memory.c | 5 +++++
> > > > 4 files changed, 24 insertions(+), 4 deletions(-)
> > > >
> > > > diff --git a/Documentation/x86/sgx.rst
> > > > b/Documentation/x86/sgx.rst
> > > > index dd0ac96ff9ef..68ee171e1d8f 100644
> > > > --- a/Documentation/x86/sgx.rst
> > > > +++ b/Documentation/x86/sgx.rst
> > > > @@ -250,3 +250,9 @@ user wants to deploy SGX applications both
> > > > on the host and in guests
> > > > on the same machine, the user should reserve enough EPC (by
> > > > taking out
> > > > total virtual EPC size of all SGX VMs from the physical EPC
> > > > size) for
> > > > host SGX applications so they can run with acceptable
> > > > performance.
> > > > +
> > > > +Supplemental fields for /proc/meminfo
> > > > +=====================================
> > > > +
> > > > +SGX_MemTotal
> > > > + The total usable SGX protected memory in kilobytes.
> > > > diff --git a/arch/x86/include/asm/sgx.h
> > > > b/arch/x86/include/asm/sgx.h
> > > > index 05f3e21f01a7..2ae9dc8c9411 100644
> > > > --- a/arch/x86/include/asm/sgx.h
> > > > +++ b/arch/x86/include/asm/sgx.h
> > > > @@ -365,6 +365,13 @@ struct sgx_sigstruct {
> > > > * comment!
> > > > */
> > > >
> > > > +#if defined(CONFIG_X86_SGX) || defined(CONFIG_X86_SGX_KVM)
> > > > +extern unsigned long sgx_nr_all_pages;
> > > > +
> > > > +int sgx_set_attribute(unsigned long *allowed_attributes,
> > > > + unsigned int attribute_fd);
> > > > +#endif
> > > > +
> > > > #ifdef CONFIG_X86_SGX_KVM
> > > > int sgx_virt_ecreate(struct sgx_pageinfo *pageinfo, void
> > > > __user *secs,
> > > > int *trapnr);
> > > > @@ -372,7 +379,4 @@ int sgx_virt_einit(void __user *sigstruct,
> > > > void __user *token,
> > > > void __user *secs, u64 *lepubkeyhash, int
> > > > *trapnr);
> > > > #endif
> > > >
> > > > -int sgx_set_attribute(unsigned long *allowed_attributes,
> > > > - unsigned int attribute_fd);
> > > > -
> > >
> > > This change seems unrelated?
> >
> > It's just a good practice not to define symbols that do not exist,
> > so that
> > if the symbol is ever used, we get a compilation error, not linking
> > error.
> >
> > Since this is included to set_memory.c, based on this conclusion, I
> > added
> > the check.
> >
>
> It would make sense to make this change in a separate patch since.
>
> thanks,
> -- Shuah
NP, I can split it.
/Jarkko
On 8/18/21 6:25 AM, Jarkko Sakkinen wrote:
> The amount of SGX memory on the system is determined by the BIOS and it
> varies wildly between systems. It can be from dozens of MB's on desktops
> or VM's, up to many GB's on servers. Just like for regular memory, it is
> sometimes useful to know the amount of usable SGX memory in the system.
>
> Add SGX_MemTotal field to /proc/meminfo, which shows the total amount of
> usable SGX memory in the system. E.g. with 32 MB reserved for SGX from
> BIOS, the printout would be:
>
> SGX_MemTotal: 22528 kB
The big question here: Do we want to put purely architecture-specific
entries in (the currently quite arch-independent) /proc/meminfo?
The current "DirectMap4k/2M/1G" entries from arch_report_meminfo() are
arch-specific in their sizes, of course, but the concept is at least
quite arch-independent.
On Mon, 2021-08-23 at 08:15 -0700, Dave Hansen wrote:
> On 8/18/21 6:25 AM, Jarkko Sakkinen wrote:
> > The amount of SGX memory on the system is determined by the BIOS and it
> > varies wildly between systems. It can be from dozens of MB's on desktops
> > or VM's, up to many GB's on servers. Just like for regular memory, it is
> > sometimes useful to know the amount of usable SGX memory in the system.
> >
> > Add SGX_MemTotal field to /proc/meminfo, which shows the total amount of
> > usable SGX memory in the system. E.g. with 32 MB reserved for SGX from
> > BIOS, the printout would be:
> >
> > SGX_MemTotal: 22528 kB
>
> The big question here: Do we want to put purely architecture-specific
> entries in (the currently quite arch-independent) /proc/meminfo?
>
> The current "DirectMap4k/2M/1G" entries from arch_report_meminfo() are
> arch-specific in their sizes, of course, but the concept is at least
> quite arch-independent.
I started with /proc/meminfo instead of /sys/devices/system/node/*/meminfo
because there is pre-existing arch callback, and secondary because it's also
better fit for my kselftest code.
The same discussion still applies to both files, they probably should follow a
matching pattern.
/Jarkko