The Nitro Enclaves driver handles the enclave lifetime management. This
includes enclave creation, termination and setting up its resources such
as memory and CPU.
An enclave runs alongside the VM that spawned it. It is abstracted as a
process running in the VM that launched it. The process interacts with
the NE driver, that exposes an ioctl interface for creating an enclave
and setting up its resources.
Include part of the KVM ioctls in the provided ioctl interface, with
additional NE ioctl commands that e.g. triggers the enclave run.
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Andra Paraschiv <[email protected]>
---
Changelog
v2 -> v3
* Remove the GPL additional wording as SPDX-License-Identifier is already in
place.
v1 -> v2
* Add ioctl for getting enclave image load metadata.
* Update NE_ENCLAVE_START ioctl name to NE_START_ENCLAVE.
* Add entry in Documentation/userspace-api/ioctl/ioctl-number.rst for NE ioctls.
* Update NE ioctls definition based on the updated ioctl range for major and
minor.
---
.../userspace-api/ioctl/ioctl-number.rst | 5 +-
include/linux/nitro_enclaves.h | 11 ++++
include/uapi/linux/nitro_enclaves.h | 65 +++++++++++++++++++
3 files changed, 80 insertions(+), 1 deletion(-)
create mode 100644 include/linux/nitro_enclaves.h
create mode 100644 include/uapi/linux/nitro_enclaves.h
diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst
index f759edafd938..8a19b5e871d3 100644
--- a/Documentation/userspace-api/ioctl/ioctl-number.rst
+++ b/Documentation/userspace-api/ioctl/ioctl-number.rst
@@ -325,8 +325,11 @@ Code Seq# Include File Comments
0xAC 00-1F linux/raw.h
0xAD 00 Netfilter device in development:
<mailto:[email protected]>
-0xAE all linux/kvm.h Kernel-based Virtual Machine
+0xAE 00-1F linux/kvm.h Kernel-based Virtual Machine
<mailto:[email protected]>
+0xAE 40-FF linux/kvm.h Kernel-based Virtual Machine
+ <mailto:[email protected]>
+0xAE 20-3F linux/nitro_enclaves.h Nitro Enclaves
0xAF 00-1F linux/fsl_hypervisor.h Freescale hypervisor
0xB0 all RATIO devices in development:
<mailto:[email protected]>
diff --git a/include/linux/nitro_enclaves.h b/include/linux/nitro_enclaves.h
new file mode 100644
index 000000000000..d91ef2bfdf47
--- /dev/null
+++ b/include/linux/nitro_enclaves.h
@@ -0,0 +1,11 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ */
+
+#ifndef _LINUX_NITRO_ENCLAVES_H_
+#define _LINUX_NITRO_ENCLAVES_H_
+
+#include <uapi/linux/nitro_enclaves.h>
+
+#endif /* _LINUX_NITRO_ENCLAVES_H_ */
diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
new file mode 100644
index 000000000000..3413352baf32
--- /dev/null
+++ b/include/uapi/linux/nitro_enclaves.h
@@ -0,0 +1,65 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ */
+
+#ifndef _UAPI_LINUX_NITRO_ENCLAVES_H_
+#define _UAPI_LINUX_NITRO_ENCLAVES_H_
+
+#include <linux/kvm.h>
+#include <linux/types.h>
+
+/* Nitro Enclaves (NE) Kernel Driver Interface */
+
+/**
+ * The command is used to get information needed for in-memory enclave image
+ * loading e.g. offset in enclave memory to start placing the enclave image.
+ *
+ * The image load metadata is an in / out data structure. It includes info
+ * provided by the caller - flags - and returns the offset in enclave memory
+ * where to start placing the enclave image.
+ */
+#define NE_GET_IMAGE_LOAD_METADATA _IOWR(0xAE, 0x20, struct image_load_metadata)
+
+/**
+ * The command is used to trigger enclave start after the enclave resources,
+ * such as memory and CPU, have been set.
+ *
+ * The enclave start metadata is an in / out data structure. It includes info
+ * provided by the caller - enclave cid and flags - and returns the slot uid
+ * and the cid (if input cid is 0).
+ */
+#define NE_START_ENCLAVE _IOWR(0xAE, 0x21, struct enclave_start_metadata)
+
+/* Metadata necessary for in-memory enclave image loading. */
+struct image_load_metadata {
+ /**
+ * Flags to determine the enclave image type e.g. Enclave Image Format
+ * (EIF) (in).
+ */
+ __u64 flags;
+
+ /**
+ * Offset in enclave memory where to start placing the enclave image
+ * (out).
+ */
+ __u64 memory_offset;
+};
+
+/* Setup metadata necessary for enclave start. */
+struct enclave_start_metadata {
+ /* Flags for the enclave to start with (e.g. debug mode) (in). */
+ __u64 flags;
+
+ /**
+ * Context ID (CID) for the enclave vsock device. If 0 as input, the
+ * CID is autogenerated by the hypervisor and returned back as output
+ * by the driver (in/out).
+ */
+ __u64 enclave_cid;
+
+ /* Slot unique id mapped to the enclave to start (out). */
+ __u64 slot_uid;
+};
+
+#endif /* _UAPI_LINUX_NITRO_ENCLAVES_H_ */
--
2.20.1 (Apple Git-117)
Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.
On Tue, May 26, 2020 at 01:13:17AM +0300, Andra Paraschiv wrote:
> The Nitro Enclaves driver handles the enclave lifetime management. This
> includes enclave creation, termination and setting up its resources such
> as memory and CPU.
>
> An enclave runs alongside the VM that spawned it. It is abstracted as a
> process running in the VM that launched it. The process interacts with
> the NE driver, that exposes an ioctl interface for creating an enclave
> and setting up its resources.
>
> Include part of the KVM ioctls in the provided ioctl interface, with
> additional NE ioctl commands that e.g. triggers the enclave run.
>
> Signed-off-by: Alexandru Vasile <[email protected]>
> Signed-off-by: Andra Paraschiv <[email protected]>
> ---
> Changelog
>
> v2 -> v3
>
> * Remove the GPL additional wording as SPDX-License-Identifier is already in
> place.
>
> v1 -> v2
>
> * Add ioctl for getting enclave image load metadata.
> * Update NE_ENCLAVE_START ioctl name to NE_START_ENCLAVE.
> * Add entry in Documentation/userspace-api/ioctl/ioctl-number.rst for NE ioctls.
> * Update NE ioctls definition based on the updated ioctl range for major and
> minor.
> ---
> .../userspace-api/ioctl/ioctl-number.rst | 5 +-
> include/linux/nitro_enclaves.h | 11 ++++
> include/uapi/linux/nitro_enclaves.h | 65 +++++++++++++++++++
> 3 files changed, 80 insertions(+), 1 deletion(-)
> create mode 100644 include/linux/nitro_enclaves.h
> create mode 100644 include/uapi/linux/nitro_enclaves.h
>
> diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst
> index f759edafd938..8a19b5e871d3 100644
> --- a/Documentation/userspace-api/ioctl/ioctl-number.rst
> +++ b/Documentation/userspace-api/ioctl/ioctl-number.rst
> @@ -325,8 +325,11 @@ Code Seq# Include File Comments
> 0xAC 00-1F linux/raw.h
> 0xAD 00 Netfilter device in development:
> <mailto:[email protected]>
> -0xAE all linux/kvm.h Kernel-based Virtual Machine
> +0xAE 00-1F linux/kvm.h Kernel-based Virtual Machine
> <mailto:[email protected]>
> +0xAE 40-FF linux/kvm.h Kernel-based Virtual Machine
> + <mailto:[email protected]>
> +0xAE 20-3F linux/nitro_enclaves.h Nitro Enclaves
> 0xAF 00-1F linux/fsl_hypervisor.h Freescale hypervisor
> 0xB0 all RATIO devices in development:
> <mailto:[email protected]>
Reusing KVM ioctls seems a little hacky. Even the ioctls that are used
by this driver don't use all the fields or behave in the same way as
kvm.ko.
For example, the memory regions slot number is not used by Nitro
Enclaves.
It would be cleaner to define NE-specific ioctls instead.
> diff --git a/include/linux/nitro_enclaves.h b/include/linux/nitro_enclaves.h
> new file mode 100644
> index 000000000000..d91ef2bfdf47
> --- /dev/null
> +++ b/include/linux/nitro_enclaves.h
> @@ -0,0 +1,11 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
> + */
> +
> +#ifndef _LINUX_NITRO_ENCLAVES_H_
> +#define _LINUX_NITRO_ENCLAVES_H_
> +
> +#include <uapi/linux/nitro_enclaves.h>
> +
> +#endif /* _LINUX_NITRO_ENCLAVES_H_ */
> diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
> new file mode 100644
> index 000000000000..3413352baf32
> --- /dev/null
> +++ b/include/uapi/linux/nitro_enclaves.h
> @@ -0,0 +1,65 @@
> +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
> +/*
> + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
> + */
> +
> +#ifndef _UAPI_LINUX_NITRO_ENCLAVES_H_
> +#define _UAPI_LINUX_NITRO_ENCLAVES_H_
> +
> +#include <linux/kvm.h>
> +#include <linux/types.h>
> +
> +/* Nitro Enclaves (NE) Kernel Driver Interface */
> +
> +/**
> + * The command is used to get information needed for in-memory enclave image
> + * loading e.g. offset in enclave memory to start placing the enclave image.
> + *
> + * The image load metadata is an in / out data structure. It includes info
> + * provided by the caller - flags - and returns the offset in enclave memory
> + * where to start placing the enclave image.
> + */
> +#define NE_GET_IMAGE_LOAD_METADATA _IOWR(0xAE, 0x20, struct image_load_metadata)
> +
> +/**
> + * The command is used to trigger enclave start after the enclave resources,
> + * such as memory and CPU, have been set.
> + *
> + * The enclave start metadata is an in / out data structure. It includes info
> + * provided by the caller - enclave cid and flags - and returns the slot uid
> + * and the cid (if input cid is 0).
> + */
> +#define NE_START_ENCLAVE _IOWR(0xAE, 0x21, struct enclave_start_metadata)
> +
image_load_metadata->flags and enclave_start_metadata->flags constants
are missing.
> +/* Metadata necessary for in-memory enclave image loading. */
> +struct image_load_metadata {
> + /**
> + * Flags to determine the enclave image type e.g. Enclave Image Format
> + * (EIF) (in).
> + */
> + __u64 flags;
> +
> + /**
> + * Offset in enclave memory where to start placing the enclave image
> + * (out).
> + */
> + __u64 memory_offset;
> +};
What about feature bits or a API version number field? If you add
features to the NE driver, how will userspace detect them?
Even if you intend to always compile userspace against the exact kernel
headers that the program will run on, it can still be useful to have an
API version for informational purposes and to easily prevent user
errors (running a new userspace binary on an old kernel where the API is
different).
Finally, reserved struct fields may come in handy in the future. That
way userspace and the kernel don't need to explicitly handle multiple
struct sizes.
On 27/05/2020 11:49, Stefan Hajnoczi wrote:
> On Tue, May 26, 2020 at 01:13:17AM +0300, Andra Paraschiv wrote:
>> The Nitro Enclaves driver handles the enclave lifetime management. This
>> includes enclave creation, termination and setting up its resources such
>> as memory and CPU.
>>
>> An enclave runs alongside the VM that spawned it. It is abstracted as a
>> process running in the VM that launched it. The process interacts with
>> the NE driver, that exposes an ioctl interface for creating an enclave
>> and setting up its resources.
>>
>> Include part of the KVM ioctls in the provided ioctl interface, with
>> additional NE ioctl commands that e.g. triggers the enclave run.
>>
>> Signed-off-by: Alexandru Vasile <[email protected]>
>> Signed-off-by: Andra Paraschiv <[email protected]>
>> ---
>> Changelog
>>
>> v2 -> v3
>>
>> * Remove the GPL additional wording as SPDX-License-Identifier is already in
>> place.
>>
>> v1 -> v2
>>
>> * Add ioctl for getting enclave image load metadata.
>> * Update NE_ENCLAVE_START ioctl name to NE_START_ENCLAVE.
>> * Add entry in Documentation/userspace-api/ioctl/ioctl-number.rst for NE ioctls.
>> * Update NE ioctls definition based on the updated ioctl range for major and
>> minor.
>> ---
>> .../userspace-api/ioctl/ioctl-number.rst | 5 +-
>> include/linux/nitro_enclaves.h | 11 ++++
>> include/uapi/linux/nitro_enclaves.h | 65 +++++++++++++++++++
>> 3 files changed, 80 insertions(+), 1 deletion(-)
>> create mode 100644 include/linux/nitro_enclaves.h
>> create mode 100644 include/uapi/linux/nitro_enclaves.h
>>
>> diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst
>> index f759edafd938..8a19b5e871d3 100644
>> --- a/Documentation/userspace-api/ioctl/ioctl-number.rst
>> +++ b/Documentation/userspace-api/ioctl/ioctl-number.rst
>> @@ -325,8 +325,11 @@ Code Seq# Include File Comments
>> 0xAC 00-1F linux/raw.h
>> 0xAD 00 Netfilter device in development:
>> <mailto:[email protected]>
>> -0xAE all linux/kvm.h Kernel-based Virtual Machine
>> +0xAE 00-1F linux/kvm.h Kernel-based Virtual Machine
>> <mailto:[email protected]>
>> +0xAE 40-FF linux/kvm.h Kernel-based Virtual Machine
>> + <mailto:[email protected]>
>> +0xAE 20-3F linux/nitro_enclaves.h Nitro Enclaves
>> 0xAF 00-1F linux/fsl_hypervisor.h Freescale hypervisor
>> 0xB0 all RATIO devices in development:
>> <mailto:[email protected]>
> Reusing KVM ioctls seems a little hacky. Even the ioctls that are used
> by this driver don't use all the fields or behave in the same way as
> kvm.ko.
>
> For example, the memory regions slot number is not used by Nitro
> Enclaves.
>
> It would be cleaner to define NE-specific ioctls instead.
Indeed, a couple of fields / parameters are not used during the KVM
ioctl calls for now.
Will adapt the logic to follow a similar model of creating VMs and
adding resources, with NE ioctls.
>
>> diff --git a/include/linux/nitro_enclaves.h b/include/linux/nitro_enclaves.h
>> new file mode 100644
>> index 000000000000..d91ef2bfdf47
>> --- /dev/null
>> +++ b/include/linux/nitro_enclaves.h
>> @@ -0,0 +1,11 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
>> + */
>> +
>> +#ifndef _LINUX_NITRO_ENCLAVES_H_
>> +#define _LINUX_NITRO_ENCLAVES_H_
>> +
>> +#include <uapi/linux/nitro_enclaves.h>
>> +
>> +#endif /* _LINUX_NITRO_ENCLAVES_H_ */
>> diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
>> new file mode 100644
>> index 000000000000..3413352baf32
>> --- /dev/null
>> +++ b/include/uapi/linux/nitro_enclaves.h
>> @@ -0,0 +1,65 @@
>> +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
>> +/*
>> + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
>> + */
>> +
>> +#ifndef _UAPI_LINUX_NITRO_ENCLAVES_H_
>> +#define _UAPI_LINUX_NITRO_ENCLAVES_H_
>> +
>> +#include <linux/kvm.h>
>> +#include <linux/types.h>
>> +
>> +/* Nitro Enclaves (NE) Kernel Driver Interface */
>> +
>> +/**
>> + * The command is used to get information needed for in-memory enclave image
>> + * loading e.g. offset in enclave memory to start placing the enclave image.
>> + *
>> + * The image load metadata is an in / out data structure. It includes info
>> + * provided by the caller - flags - and returns the offset in enclave memory
>> + * where to start placing the enclave image.
>> + */
>> +#define NE_GET_IMAGE_LOAD_METADATA _IOWR(0xAE, 0x20, struct image_load_metadata)
>> +
>> +/**
>> + * The command is used to trigger enclave start after the enclave resources,
>> + * such as memory and CPU, have been set.
>> + *
>> + * The enclave start metadata is an in / out data structure. It includes info
>> + * provided by the caller - enclave cid and flags - and returns the slot uid
>> + * and the cid (if input cid is 0).
>> + */
>> +#define NE_START_ENCLAVE _IOWR(0xAE, 0x21, struct enclave_start_metadata)
>> +
> image_load_metadata->flags and enclave_start_metadata->flags constants
> are missing.
I added them now in this file. Thank you.
>
>> +/* Metadata necessary for in-memory enclave image loading. */
>> +struct image_load_metadata {
>> + /**
>> + * Flags to determine the enclave image type e.g. Enclave Image Format
>> + * (EIF) (in).
>> + */
>> + __u64 flags;
>> +
>> + /**
>> + * Offset in enclave memory where to start placing the enclave image
>> + * (out).
>> + */
>> + __u64 memory_offset;
>> +};
> What about feature bits or a API version number field? If you add
> features to the NE driver, how will userspace detect them?
>
> Even if you intend to always compile userspace against the exact kernel
> headers that the program will run on, it can still be useful to have an
> API version for informational purposes and to easily prevent user
> errors (running a new userspace binary on an old kernel where the API is
> different).
Good point. I'll add a get API version ioctl for this purpose.
>
> Finally, reserved struct fields may come in handy in the future. That
> way userspace and the kernel don't need to explicitly handle multiple
> struct sizes.
True, I've seen this pattern of including reserved fields e.g. in a
couple of KVM data structures.
Thanks for feedback, Stefan.
Andra
Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.
On Wed, 2020-05-27 at 09:49 +0100, Stefan Hajnoczi wrote:
>
> What about feature bits or a API version number field? If you add
> features to the NE driver, how will userspace detect them?
>
> Even if you intend to always compile userspace against the exact kernel
> headers that the program will run on, it can still be useful to have an
> API version for informational purposes and to easily prevent user
> errors (running a new userspace binary on an old kernel where the API is
> different).
>
> Finally, reserved struct fields may come in handy in the future. That
> way userspace and the kernel don't need to explicitly handle multiple
> struct sizes.
Beware, Greg might disagree :)
That said, yes, at least a way to query the API version would be
useful.
Cheers,
Ben.
On 01/06/2020 06:02, Benjamin Herrenschmidt wrote:
> On Wed, 2020-05-27 at 09:49 +0100, Stefan Hajnoczi wrote:
>> What about feature bits or a API version number field? If you add
>> features to the NE driver, how will userspace detect them?
>>
>> Even if you intend to always compile userspace against the exact kernel
>> headers that the program will run on, it can still be useful to have an
>> API version for informational purposes and to easily prevent user
>> errors (running a new userspace binary on an old kernel where the API is
>> different).
>>
>> Finally, reserved struct fields may come in handy in the future. That
>> way userspace and the kernel don't need to explicitly handle multiple
>> struct sizes.
> Beware, Greg might disagree :)
>
> That said, yes, at least a way to query the API version would be
> useful.
I see there are several thoughts with regard to extensions possibilities. :)
I added an ioctl for getting the API version, we have now a way to query
that info. Also, I updated the sample in this patch series to check for
the API version.
Thanks,
Andra
Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.
On Mon, Jun 01, 2020 at 10:20:18AM +0300, Paraschiv, Andra-Irina wrote:
>
>
> On 01/06/2020 06:02, Benjamin Herrenschmidt wrote:
> > On Wed, 2020-05-27 at 09:49 +0100, Stefan Hajnoczi wrote:
> > > What about feature bits or a API version number field? If you add
> > > features to the NE driver, how will userspace detect them?
> > >
> > > Even if you intend to always compile userspace against the exact kernel
> > > headers that the program will run on, it can still be useful to have an
> > > API version for informational purposes and to easily prevent user
> > > errors (running a new userspace binary on an old kernel where the API is
> > > different).
> > >
> > > Finally, reserved struct fields may come in handy in the future. That
> > > way userspace and the kernel don't need to explicitly handle multiple
> > > struct sizes.
> > Beware, Greg might disagree :)
> >
> > That said, yes, at least a way to query the API version would be
> > useful.
>
> I see there are several thoughts with regard to extensions possibilities. :)
>
> I added an ioctl for getting the API version, we have now a way to query
> that info. Also, I updated the sample in this patch series to check for the
> API version.
Great. The ideas are orthogonal and not all of them need to be used
together. As long as their is a way of extending the API cleanly in the
future then extensions can be made without breaking userspace.
Stefan
On 05/06/2020 11:15, Stefan Hajnoczi wrote:
> On Mon, Jun 01, 2020 at 10:20:18AM +0300, Paraschiv, Andra-Irina wrote:
>>
>> On 01/06/2020 06:02, Benjamin Herrenschmidt wrote:
>>> On Wed, 2020-05-27 at 09:49 +0100, Stefan Hajnoczi wrote:
>>>> What about feature bits or a API version number field? If you add
>>>> features to the NE driver, how will userspace detect them?
>>>>
>>>> Even if you intend to always compile userspace against the exact kernel
>>>> headers that the program will run on, it can still be useful to have an
>>>> API version for informational purposes and to easily prevent user
>>>> errors (running a new userspace binary on an old kernel where the API is
>>>> different).
>>>>
>>>> Finally, reserved struct fields may come in handy in the future. That
>>>> way userspace and the kernel don't need to explicitly handle multiple
>>>> struct sizes.
>>> Beware, Greg might disagree :)
>>>
>>> That said, yes, at least a way to query the API version would be
>>> useful.
>> I see there are several thoughts with regard to extensions possibilities. :)
>>
>> I added an ioctl for getting the API version, we have now a way to query
>> that info. Also, I updated the sample in this patch series to check for the
>> API version.
> Great. The ideas are orthogonal and not all of them need to be used
> together. As long as their is a way of extending the API cleanly in the
> future then extensions can be made without breaking userspace.
Agree, as we achieve the ultimate goal of having a stable interface,
open for extensions without breaking changes.
Thanks,
Andra
Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.