Message-ID: <c0aab85a-f6a1-ed5c-5540-b03778ffe24a@intel.com>
Date:   Fri, 3 Dec 2021 14:12:17 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Firefox/91.0 Thunderbird/91.3.2
Subject: Re: [PATCH 05/25] x86/sgx: Introduce runtime protection bits
Content-Language: en-US
To:     Andy Lutomirski <luto@kernel.org>, <dave.hansen@linux.intel.com>,
        <jarkko@kernel.org>, <tglx@linutronix.de>, <bp@alien8.de>,
        <mingo@redhat.com>, <linux-sgx@vger.kernel.org>, <x86@kernel.org>
CC:     <seanjc@google.com>, <kai.huang@intel.com>,
        <cathy.zhang@intel.com>, <cedric.xing@intel.com>,
        <haitao.huang@intel.com>, <mark.shanahan@intel.com>,
        <hpa@zytor.com>, <linux-kernel@vger.kernel.org>
References: <cover.1638381245.git.reinette.chatre@intel.com>
 <2f6b04dd8949591ee6139072c72eb93da3dd07b0.1638381245.git.reinette.chatre@intel.com>
 <db9b7bc9-fdca-4dd2-2c3f-3b7354c165bb@kernel.org>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <db9b7bc9-fdca-4dd2-2c3f-3b7354c165bb@kernel.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?TmpTUzdMamZ6RWU4L3g2V0JXaFVkUnJHRll4VkhzMmdSU1B2ZXhzY1phS0o0?=
 =?utf-8?B?ZGZpSWg1MTJnUElGKy83TFYrbXRXVHpWOGYzMnN6SFBtcHJzMllZdC9qK2RP?=
 =?utf-8?B?NWkwTnlwNWlxVG0wWXhVQytia2FCUC9lZTMzeDRZZ0RtVWlqbU9wNGxoeFhD?=
 =?utf-8?B?UExIcHpOYVJ5WnBiOTY5MGxPaDNobndLTzh6WUlkT3pidkFlQVNVVWtPaGJi?=
 =?utf-8?B?TkZmZytyOXg3dTl2OFcvN2ZYRE1XRFVHTXNqNmFmQXZaT2daV1NuNUxFSkRI?=
 =?utf-8?B?Q3dZNUkyMFdvdllYK2tmbC9PdDJ1Wm90MU12ZWthWm1VenVScmlXVWdYQnF2?=
 =?utf-8?B?M1hGUnZDZVlVTkQ4NCswQ3BxSjQ0TUdYL1lrMWtRbWUzSXFwby82MGxGbGNx?=
 =?utf-8?B?QTM1cmU4V3ZtcFhwNG1NYmNIL0M0UjdldkEyTlVrajhGRGdyM1FxWUgrRktM?=
 =?utf-8?B?cjBTcTNnSFVtNTk4eHBFMGp1UmNUS3crUC8zREUwOEFRbzVTbXFFMGVLc214?=
 =?utf-8?B?cktYRUNCd0MvQnJMRVovaU0xaWZrM2pWbGtTWDVYR2l3dXJQQzc1anBHQk9K?=
 =?utf-8?B?QXdKdUI0ZWFGaEdCeG5wQS85VkNaQ0d2bFJzSmxseGNKclRGR2MxSHVQam1n?=
 =?utf-8?B?N3M2UCtWajhoaXVNcitXbUorTnhyb2RoZm9lZ20wWUhoaDdHckpYYUZpOWEr?=
 =?utf-8?B?aVFtOWFzL3ZEcmhqajZScnIzVUFLcFlLU2NGQ1JmUmp4dGNEL0hpSTBlb0xm?=
 =?utf-8?B?SjhDOERkQ1Z4bnpKZlRkbDdLaVlBdjVUNXREUFpQWC9PcnVoN1RPNnVaWFZI?=
 =?utf-8?B?aFkxcGdYcjFmTzFxVXpRWWtiNmI1YUVHamdqbEs4RE9HR0c2bUg5L2RMd3NO?=
 =?utf-8?B?QldyYU1GR2Z2WEZNbjZ2WU5haGF3NW85OTBRcUQxcjE0RVFNWG9uYityVVVx?=
 =?utf-8?B?OTFKUmJMc3ZkRWgxdzAyeXVYdzlhNWUvckphNk1yMTI4czBCcTRBL0JkUjNF?=
 =?utf-8?B?TUs0Wlh6M2NQNVQ5RUt1d1dpaGJ2V0tJMEtoeHc0SWxKY3BYMkd4bENBT3la?=
 =?utf-8?B?Mk82dytndnYwNWp6UEhkOFBwWi9pODFPQzZKTFVyOFBjYmdSRUZaMXZFQ1A2?=
 =?utf-8?B?UzhSd01xK1FBVUswZXEzSXFZQ0h0aVRaQlZaSVJsQnhzaGNZVk9Ua2J0VWYv?=
 =?utf-8?B?NG9HcjdpVnFrUHZWcHBWSFF5dzlwaG9YNmw0RFRwNUxaMGxyOUg5OVJkSjhn?=
 =?utf-8?B?TjN5R0YxUGN1L3VWaEk2dGoxSk8zMDFCTWRnLzdEeWE3MStPNGVDd1c5UjN6?=
 =?utf-8?B?U0FZYWJVSUFnTG8zdmp5TDNHcSsrOTBCT1lESUhXT0I5NnkwTXhRdzl3eDJv?=
 =?utf-8?B?MFI1alNHbG9zUlJQNlpKOGRwN0p1eGVkYmJ1K1ZKeVA2MjlKcGhuUHlUWmE0?=
 =?utf-8?B?WWJCWnBWT3hGRFlNSjNyWEZ3VXE3Q0dzTHZkU0ljd085eFJ5SGdnQnNlbUV4?=
 =?utf-8?B?YWh3UVJNbXV3Y3pQZVpsWnZZWjU5QUVkRThpM0tQUXFONDlhT3A1OENsR0d3?=
 =?utf-8?B?eWh4MFJSdW93bmpsc3h0UUtJcXphckZSbkRjSWI4c0o0aGhNYVdHQlFMYktR?=
 =?utf-8?B?bFVzVEhzemRudnRlejIxTHdoS2lZV3BmNENvaDBnQmJ5MU10OXlDUng1dk1r?=
 =?utf-8?B?dzJ0V0ZmUENsUWk3KzVMYk1JNjczanZ6RUxkb3B1VjdhRHQxNEFKaXpTY0Zw?=
 =?utf-8?B?NG53b0c2TjlxWkhwR0F1ZzhJZFgveEJBdE9qWjIxczdEeTdiUHYvTW5PNENy?=
 =?utf-8?B?N0Y4U2xhTTRVMVZJbHJLNkNMaG1jVTNkVnF3R2hxWlZNVUg0UC9MckRnRnpu?=
 =?utf-8?B?RWZHV01EQWVzVjdGYXRVTFFFenF0ZlVVVkxKTGhlM0JnL1hNUHRtQnRrWGYw?=
 =?utf-8?B?RmZpZ250WFNkdk5UT1RyalNXNjlsREVWS25XTG85UGJVM09ETGtTRnBKMmNK?=
 =?utf-8?B?TEJpa3VVQ2Z1cW5zR0dYd2pGclFNL2UwbTJid1BHYmwzblhpbXk1MlM2aS9z?=
 =?utf-8?B?N25HQiswa2Y4c2wrUjRKT2tnTjlnN1MzOEFJZUFmT0pqK01tUms4UW9NSzVK?=
 =?utf-8?B?UUo1aWRGM2xERkVtRmxLT1o0aXloQzdLZ0d6Qy9oWkxPbkwxMTgvOXpXS1RO?=
 =?utf-8?B?b3AxekhERkI3c3hVOUN0OHBnNldIS2tuUVdScEUyRXVHNGxnRWpIcXk2SkM2?=
 =?utf-8?Q?KhUpovZyYAZiR1O1fwBKbRMf2GeYtO3wg0emzfZvQ4=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 1f7fd571-b773-472d-888f-08d9b6a9faa2
X-MS-Exchange-CrossTenant-AuthSource: BN0PR11MB5744.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Dec 2021 22:12:22.4212
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: rsYO68rZiG0Tn4RA5xmSdsG770dqlaZ2ScbL8q19V6n+w9rNCDc7DQmZH1oklBiVJgCvysdCel2MATuu84Biiqj8yStQQ93jKZqbXBYUdHo=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR11MB4114
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Andy,

On 12/3/2021 11:28 AM, Andy Lutomirski wrote:
> On 12/1/21 11:23, Reinette Chatre wrote:
>> Enclave creators declare their paging permission intent at the time
>> the pages are added to the enclave. These paging permissions are
>> vetted when pages are added to the enclave and stashed off
>> (in sgx_encl_page->vm_max_prot_bits) for later comparison with
>> enclave PTEs.
>>
> 
> I'm a bit confused here. ENCLU[EMODPE] allows the enclave to change the 
> EPCM permission bits however it likes with no oversight from the kernel. 
>   So we end up with a whole bunch of permission masks:

Before jumping to the permission masks I would like to step back and 
just confirm the context. We need to consider the following three 
permissions:

EPCM permissions: the enclave page permissions maintained in the SGX 
hardware. The OS is constrained here in that it cannot query the current 
EPCM permissions. Even so, the OS needs to ensure PTEs are installed 
appropriately (we do not want a RW PTE for a read-only enclave page) and 
thus the OS keeps its own record of EPCM permissions to support this.
As you note later even in current kernel the enclave can change these 
permissions without OS knowing. EPCM permissions can only be relaxed 
without the OS knowledge though so the OS record of EPCM permissions can 
only ever be stricter than the actual EPCM permissions.

VMA permissions: Current behavior (not changed in this series) is that 
the OS enforces that a new VMA should have the same or weaker 
permissions than the EPCM permissions.

Page table entries: These should match the EPCM permissions without 
exceeding VMA permissions.

> The PTE: controlled by complex kernel policy
> 
> The VMA: with your series, this is entirely controlled by userspace.  I 
> think I'm fine with that.
> 
> vm_max_prot_bits: populated from secinfo at setup time, unless I missed 
> something that changes it later.  Maybe I'm confused or missed something 
> in one of the patches,

Yes, vm_max_prot_bits is currently and continues to be populated from 
secinfo for pages added before the enclave is initialized and in a later 
patch it would be hardcoded to RW for pages that are added after the 
enclave is initialized.  In the current implementation vm_max_prot_bits 
is the OS's record of the EPCM permissions used to guide VMA and PTE 
permissions.

On a higher level, the implementation decision is that vm_max_prot_bits 
is the static "vetted" permissions of a page - the maximum permissions a 
page is allowed to have during its entire lifetime. This matches the 
current implementation. In the current implementation permissions are 
only able to change via VMA and PTE ... for example a read-only VMA can 
access an enclave page with vm_max_prot_bits of RW. With the SGX2 
support permission changes are allowed to EPCM permissions - but in this 
implementation they are not allowed to exceed the originally vetted 
vm_max_prot_bits.

In this SGX2 implementation an enclave page could thus be added to an 
enclave with secinfo and vm_max_prot_bits of RW that would only allow 
that page to have R or RW permissions (VMA, PTE, and OS view of EPCM 
permissions) in its lifetime, never RX or RWX. Yes, it may be possible 
for the enclave to change the EPCM permissions from within the enclave 
using ENCLU[EMODPE] but to access the page the enclave would need the OS 
to install the appropriate PTE and the OS would not do so if 
vm_max_prot_bits does not allow it. Neither would the OS allow an 
executable VMA.

> 
> vm_run_prot_bits: populated from some combination of ioctls.  I'm 
> entirely lost as to what this is for.

With SGX2 it is possible to change the EPCM permissions of an enclave 
page after the enclave is initialized. vm_max_prot_bits would provide 
guidance to what permissions a page is allowed to have while 
vm_run_prot_bits contains the current view of EPCM permissions used by 
the OS to guide whether requested VMA permissions are allowed and guide 
what PTE permissions should be.

Consider this example how vm_max_prot_bits and vm_run_prot_bits are used:

(1) Add enclave page with secinfo of RW to uninitialized enclave
     vm_max_prot_bits = RW
     vm_run_prot_bits = RW

(2) User space runs SGX_IOC_PAGE_MODP to change the permissions to read-
     only. This is allowed because vm_max_prot_bits = RW. Now:
     vm_max_prot_bits = RW
     vm_run_prot_bits = R

     Now VMAs are created and PTEs installed based on value of
     vm_run_prot_bits - write access will not be allowed.

(3) User space runs SGX_IOC_PAGE_MODP to change the permissions to RX.
     This will be denied because vm_max_prot_bits = RW.

(3) User space runs SGX_IOC_PAGE_MODP to change the permissions to RW.
     This will be allowed because vm_max_prot_bits = RW.

     Now VMAs are created and PTEs installed based on value of
     vm_run_prot_bits - write access will again be allowed.

> 
> EPCM bits: controlled by the guest.  basically useless for any host 
> purpose on SGX2 hardware (with or without kernel support -- the enclave 
> can do ENCLU[EMODPE] whether we like it or not, even on old kernels)

Indeed - permissions can only be relaxed without the OS knowledge so the 
OS's view would always be the same or stricter than the enclave.

> So I guess I don't understand the purpose of this patch    or of the 
> rules in the later patches, and I feel like this is getting more 
> complicated than makes sense.
> 
> 
> Could we perhaps make vm_max_prot_bits dynamic or at least controllable 
> in some useful way?  My initial proposal (years ago) was for 
> vm_max_prot_bits to be *separately* configured at initial load time 
> instead of being inferred from secinfo with the intent being that the 
> user untrusted runtime would set it appropriately.  I have no problem 
> with allowing runtime changes as long as the security policy makes sense 
> and it's kept consistent with PTEs.

This SGX2 enabling indeed builds on the current implementation where 
vm_max_prot_bits is inferred from secinfo. The intent is for 
vm_max_prot_bits to reflect the maximum allowed vetted permissions.

At this time vm_max_prot_bits is indeed static and thus creates the need 
for (dynamic) vm_run_prot_bits that reflects the current EPCM 
permissions and guides VMA and PTE permissions while vm_max_prot_bits 
guides new permission requests. From what I understand this 
implementation follows the current security policy - permissions are 
never allowed to exceed the originally vetted permissions. PTEs are kept 
consistent in that they match the (vetted, OS view of) EPCM permissions.

> Also, I think we need a changelog message or, even better, actual docs 
> in kernel, explaining the actual final set of rules and invariants for 
> all these masks.

I will add a section to Documentation/x86/sgx.rst.

Reinette