Received: by 2002:a25:c205:0:0:0:0:0 with SMTP id s5csp3912896ybf; Tue, 3 Mar 2020 15:40:15 -0800 (PST) X-Google-Smtp-Source: ADFU+vv3yIRjTsCAi/HNmCqOXU3umoEQs/b/1QbO0kbDW3r2UlNYPps48hKNGm/vPIG+kZ3e78xU X-Received: by 2002:a9d:6ad1:: with SMTP id m17mr294897otq.198.1583278814863; Tue, 03 Mar 2020 15:40:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1583278814; cv=none; d=google.com; s=arc-20160816; b=hNaWLa1bjpQchmZH8xq8atSqrpSSGZtSsRvxGq8oubXDlFCiI9N7yUIBU1fwCtCmQm tt7iHKDpnNNUQZmAuu4wHA4amdTgNFDp7jPwKeCv4Z34t419t5gpb2mXFfkG9O+OxNg7 yTN7ImChmP2xkjmASSe6PZuRDPXx85lWga/3KQdQ6g/ZI8isNbqqgMYJhGpOJ68/sX6u r8CWNnew0/WyAvHnURfzUAJN9xaN5fDrz/HHT523IZYJpgUx0B1HKsvl/MDLeI8g/hm5 FLl7NXN3RL2TenFaO05SJt4WSERHIUWV5TYpCcUhpMj/0+XvnoemzHgehbut8Sief74M Lzmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=opr5Y8BdoWIHfPEAPqnXOErk4wcM905SuS2vzvFCs1M=; b=cHuOG+mWBZpLg/0G3dq97bS5wB3pANjTTRBc7LqLCKB+UFGPPKfy3JVcap0y8ko1hy PJ8Q7ZYSNAhJdMcutnw0HF2D7fr0Wzsfg9+dMOhK09/b1oQXSPNKWQsxAekJJ2mDu/75 bjJBnGUxGNtRJUATyCG6RwdjVa4RV1od9rD0e0rmbbJgZgjsTrRnoeWZuGkB4XH1iNJf Qkl9luY4SU7ALUllDN+I/c9l9PGIvi9bIQBBEffrHHDKCWFn7AgY6BdhjqHTYDdOZFaI wF17YIQrBiwyL85mMyKh1C8+ki1HcmNE5APgABEzCpnxebGZc0/SCHT6jSMsba34mpvu I/9A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id j11si138661oie.254.2020.03.03.15.40.03; Tue, 03 Mar 2020 15:40:14 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728588AbgCCXin (ORCPT + 99 others); Tue, 3 Mar 2020 18:38:43 -0500 Received: from mga03.intel.com ([134.134.136.65]:44616 "EHLO mga03.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728373AbgCCXin (ORCPT ); Tue, 3 Mar 2020 18:38:43 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 03 Mar 2020 15:38:42 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,511,1574150400"; d="scan'208";a="229117672" Received: from kwasilew-mobl.ger.corp.intel.com (HELO localhost) ([10.251.88.57]) by orsmga007.jf.intel.com with ESMTP; 03 Mar 2020 15:38:33 -0800 From: Jarkko Sakkinen To: linux-kernel@vger.kernel.org, x86@kernel.org, linux-sgx@vger.kernel.org Cc: akpm@linux-foundation.org, dave.hansen@intel.com, sean.j.christopherson@intel.com, nhorman@redhat.com, npmccallum@redhat.com, haitao.huang@intel.com, andriy.shevchenko@linux.intel.com, tglx@linutronix.de, kai.svahn@intel.com, bp@alien8.de, josh@joshtriplett.org, luto@kernel.org, kai.huang@intel.com, rientjes@google.com, cedric.xing@intel.com, puiterwijk@redhat.com, Jarkko Sakkinen , linux-doc@vger.kernel.org, Randy Dunlap Subject: [PATCH v28 12/22] docs: x86/sgx: Document SGX micro architecture and kernel internals Date: Wed, 4 Mar 2020 01:35:59 +0200 Message-Id: <20200303233609.713348-13-jarkko.sakkinen@linux.intel.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200303233609.713348-1-jarkko.sakkinen@linux.intel.com> References: <20200303233609.713348-1-jarkko.sakkinen@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=a Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Document Intel SGX micro architecture and kernel internals. The motivation is to make the core ideas approachable by keeping a fairly high abstraction level. Fine-grained micro architecture details can be looked up from Intel SDM Volume 3D. Cc: linux-doc@vger.kernel.org Co-developed-by: Sean Christopherson Signed-off-by: Sean Christopherson Acked-by: Randy Dunlap Signed-off-by: Jarkko Sakkinen --- Documentation/x86/index.rst | 1 + Documentation/x86/sgx.rst | 192 ++++++++++++++++++++++++++++++++++++ 2 files changed, 193 insertions(+) create mode 100644 Documentation/x86/sgx.rst diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst index a8de2fbc1caa..971f30a7d166 100644 --- a/Documentation/x86/index.rst +++ b/Documentation/x86/index.rst @@ -31,3 +31,4 @@ x86-specific Documentation usb-legacy-support i386/index x86_64/index + sgx diff --git a/Documentation/x86/sgx.rst b/Documentation/x86/sgx.rst new file mode 100644 index 000000000000..740b09323f18 --- /dev/null +++ b/Documentation/x86/sgx.rst @@ -0,0 +1,192 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============ +Architecture +============ + +*Software Guard eXtensions (SGX)* is a set of instructions that enable ring-3 +applications to set aside private regions of code and data. These regions are +called enclaves. An enclave can be entered to a fixed set of entry points. Only +a CPU running inside the enclave can access its code and data. + +The support can be determined by + + ``grep sgx /proc/cpuinfo`` + +Enclave Page Cache +================== + +SGX utilizes an *Enclave Page Cache (EPC)* to store pages that are associated +with an enclave. It is contained in a BIOS reserved region of physical memory. +Unlike pages used for regular memory, pages can only be accessed outside the +enclave for different purposes with the instructions **ENCLS**, **ENCLV** and +**ENCLU**. + +Direct memory accesses to an enclave can be only done by a CPU executing inside +the enclave. An enclave can be entered with **ENCLU[EENTER]** to a fixed set of +entry points. However, a CPU executing inside the enclave can do outside memory +accesses. + +Page Types +---------- + +**SGX Enclave Control Structure (SECS)** + Enclave's address range, attributes and other global data are defined + by this structure. + +**Regular (REG)** + Regular EPC pages contain the code and data of an enclave. + +**Thread Control Structure (TCS)** + Thread Control Structure pages define the entry points to an enclave and + track the execution state of an enclave thread. + +**Version Array (VA)** + Version Array pages contain 512 slots, each of which can contain a version + number for a page evicted from the EPC. + +Enclave Page Cache Map +---------------------- + +The processor tracks EPC pages via the *Enclave Page Cache Map (EPCM)*. EPCM +contains an entry for each EPC page, which describes the owning enclave, access +rights and page type among the other things. + +The permissions from EPCM is consulted if and only if walking the kernel page +tables succeeds. The total permissions are thus a conjunction between page table +and EPCM permissions. + +For all intents and purposes the SGX architecture allows the processor to +invalidate all EPCM entries at will, i.e. requires that software be prepared to +handle an EPCM fault at any time. The contents of EPC are encrypted with an +ephemeral key, which is lost on power transitions. + +EPC management +============== + +EPC pages do not have ``struct page`` instances. They are IO memory from kernel +perspective. The consequence is that they are always mapped as shared memory. +Kernel defines ``/dev/sgx/enclave`` that can be mapped as ``MAP_SHARED`` to +define the address range for an enclave. + +EPC Over-subscription +===================== + +When the amount of free EPC pages goes below a low watermark the swapping thread +starts reclaiming pages. The pages that do not have the **A** bit set are +selected as victim pages. + +Launch Control +============== + +SGX provides a launch control mechanism. After all enclave pages have been +copied, kernel executes **ENCLS[EINIT]**, which initializes the enclave. Only +after this the CPU can execute inside the enclave. + +This leaf function takes an RSA-3072 signature of the enclave measurement and an +optional cryptographic token. Linux does not take advantage of launch tokens. +The instruction checks that the signature is signed with the key defined in +**IA32_SGXLEPUBKEYHASH?** MSRs and the measurement is correct. If so, the +enclave is allowed to be executed. + +MSRs can be configured by the BIOS to be either readable or writable. Linux +supports only writable configuration in order to give full control to the kernel +on launch control policy. Readable configuration requires the use of previously +mentioned launch tokens. + +The current kernel implementation supports only writable MSRs. The launch is +performed by setting the MSRs to the hash of the enclave signer's public key. +The alternative would be to have *a launch enclave* that would be signed with +the key set into MSRs, which would then generate launch tokens for other +enclaves. This would only make sense with read-only MSRs, and thus the option +has been discarded. + +Attestation +=========== + +Local Attestation +----------------- + +In local attestation an enclave creates a **REPORT** data structure with +**ENCLS[EREPORT]**, which describes the origin of an enclave. In particular, it +contains a AES-CMAC of the enclave contents signed with a report key unique to +each processor. All enclaves have access to this key. + +This mechanism can also be used in addition as a communication channel as the +**REPORT** data structure includes a 64-byte field for variable information. + +Remote Attestation +------------------ + +For remote attestation (aka provisioning) there are multiple options available: + +* EPID based scheme, which requires the use of Intel managed attestation + service. +* ECDSA based scheme, which allows a 3rd party to act as an attestation service. + +Intel provides an open source *quoting enclave (QE)* and *provisioning +certification enclave (PCE)* for the ECDSA based scheme. PCE acts as the +CA for the local QE's. + +Intel also provides a proprietary binary version of the PCE. This is a +necessity when the software needs to prove to be running inside a legit enclave +on real hardware. + +The use of remote attestation must be strictly controlled because it allows to +get access to the provisioning keys to attest to a remote party that the +software is running inside a legitimate enclave on real hardware. This could be +potentially used by malware, and thus must be protected. + +Enclaves can attest their identity when **ATTRIBUTES.PROVISIONKEY** is set in +SECS. This attribute authorizes **ENCLS[EGETKEY]** to access provisioning keys. + +References +---------- + +"Intel® Software Guard Extensions: EPID Provisioning and Attestation Services" + https://software.intel.com/sites/default/files/managed/57/0e/ww10-2016-sgx-provisioning-and-attestation-final.pdf + +"Supporting Third Party Attestation for Intel® SGX with Intel® Data Center +Attestation Primitives" + https://software.intel.com/sites/default/files/managed/f1/b8/intel-sgx-support-for-third-party-attestation.pdf + +Usage Models +============ + +Shared Library +-------------- + +Sensitive data and the code that acts on it is partitioned from the application +into a separate library. The library is then linked as a DSO which can be loaded +into an enclave. The application can then make individual function calls into +the enclave through special SGX instructions. A run-time within the enclave is +configured to marshal function parameters into and out of the enclave and to +call the correct library function. + +Application Container +--------------------- + +An application may be loaded into a container enclave which is specially +configured with a library OS and run-time which permits the application to run. +The enclave run-time and library OS work together to execute the application +when a thread enters the enclave. + +================ +Kernel internals +================ + +An enclave is created by opening ``/dev/sgx/enclave`` and calling a set of ioctl +calls, which reserve a fixed range of memory addresses for the enclave and +initialize its memory contents. + +An enclave can be made visible with ``mmap()`` calls. Permissions are capped by +enclave page permissions given during the building phase because CPU disallows a +PTE have higher permissions than the enclave page that it contains. + +Enclaves can be forked or sent through UDS sockets, which allows an enclave +consumer and a builder to be separate processes with a different set of +privileges. + +The backing memory is implemented with a private shemm file, which is not +accounted. This makes it advicable to not allow all processes in a system +to build enclaves. -- 2.25.0