Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
Message-ID: <e1c04077-0165-c5ec-53be-7fd732965e80@intel.com>
Date:   Fri, 11 Mar 2022 11:28:27 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Firefox/91.0 Thunderbird/91.6.1
Subject: Re: [PATCH V2 16/32] x86/sgx: Support restricting of enclave page
 permissions
Content-Language: en-US
To:     Jarkko Sakkinen <jarkko@kernel.org>
CC:     Haitao Huang <haitao.huang@linux.intel.com>,
        "Dhanraj, Vijay" <vijay.dhanraj@intel.com>,
        "dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        "bp@alien8.de" <bp@alien8.de>,
        "Lutomirski, Andy" <luto@kernel.org>,
        "mingo@redhat.com" <mingo@redhat.com>,
        "linux-sgx@vger.kernel.org" <linux-sgx@vger.kernel.org>,
        "x86@kernel.org" <x86@kernel.org>,
        "Christopherson,, Sean" <seanjc@google.com>,
        "Huang, Kai" <kai.huang@intel.com>,
        "Zhang, Cathy" <cathy.zhang@intel.com>,
        "Xing, Cedric" <cedric.xing@intel.com>,
        "Huang, Haitao" <haitao.huang@intel.com>,
        "Shanahan, Mark" <mark.shanahan@intel.com>,
        "hpa@zytor.com" <hpa@zytor.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
References: <cover.1644274683.git.reinette.chatre@intel.com>
 <4ce06608b5351f65f4e6bc6fc87c88a71215a2e7.1644274683.git.reinette.chatre@intel.com>
 <YhLhoMFPyOFZ2fsX@iki.fi>
 <DM8PR11MB55917F499CDF4CC7D426B0A7F63C9@DM8PR11MB5591.namprd11.prod.outlook.com>
 <YimWaAqEnXHbLdjh@iki.fi> <op.1itu5vkewjvjmi@hhuan26-mobl1.mshome.net>
 <Yis8LV99mORcLYs6@iki.fi> <Yis9rA8uC/0bmWCF@iki.fi>
 <97565fed-dc67-bab1-28d4-c40201c9f055@intel.com> <YiuQx+X9UQ2l22un@iki.fi>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <YiuQx+X9UQ2l22un@iki.fi>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UWJlRmxHYnpxUllpVVByRUI0Kzd0WnlucUU5R0M5UEpkMlFsakcwbDJqeTBX?=
 =?utf-8?B?RUxBcVY4S0JQNkY2OW1rbzFJbDdwM0M0RUY5OWZzVnhRdHFrK1ViazJkS3FJ?=
 =?utf-8?B?UHRuVXFkWW9qZmxzeXRoUHV3NEhsMmZBcnFLOVlFTjVOemViUDBmRmZzTnRi?=
 =?utf-8?B?Q1dxZDF5ZHlHZGdkYW1zdHdvTUlta3l6LyszbTM0aHJzNnI4Rm5wcU0vNVVS?=
 =?utf-8?B?Njh6bXRYL1dTeUlxNXNTamdCczlTaXNPZ2RTVGJ3RVdqb2kyRmlpMWpoMzJP?=
 =?utf-8?B?WGZTem9sMFZ3ZDZOS0Jrc3dIUGtvcHdvYlpMci9ibGh3d2lQTlZnZHB4d2J5?=
 =?utf-8?B?OExPam5wVWU5dERrdjFoc0tOQmhHb0NMeFY2aVQwbEFOSzhqbmkzS1E0NjhK?=
 =?utf-8?B?aXRJNlIyVE5NWjRBQUJrSmYyOHlUVU9RSU1Ca0FyZEdCemVVWFMrcjBZMzdV?=
 =?utf-8?B?THQ5VGVmRGcrMkU0cFBTWEF0cVVoVitDNS8yUVhab0RpKzZGZmE4Tm5OUFUv?=
 =?utf-8?B?VGVtMkk1YjdhamFtYndzUHQzREVlVE5YTDFWTk1SN1JtV01LM2ZRaHphU3RW?=
 =?utf-8?B?SnRzeWIrL2dqZXllcU5IVVF6bDA4MVJkUEE0NHZkMTdIUWZoY25Id3NmVzM5?=
 =?utf-8?B?aEMrOUJLWmRlMU9OUFl2YWxvaEVKbzYraHlTSy9YSUR0NkdIYWxxcHBVbk5P?=
 =?utf-8?B?cjhGWmZPeEM0MGJSNTJ0ckZIMGxOY3d0V3BKT0Jnd0YxNUZOZXp4N2FkTWhU?=
 =?utf-8?B?dmFJU2VTZVdPU3BKWXlFK0lHZWFhTGZueW4zaDcxRmpMUGtNSWNYRWxtWkJs?=
 =?utf-8?B?TlNVSEU1aXcyRC9zQkVNK0ljbjZ6LzdaZmg5d3lZdUx3U0hYMnFiU1BQbDEv?=
 =?utf-8?B?cHJwVjVTUG1adlJvd1R4OVhac2FURkxLek4yeXc0ZEt1VnpQTnNBekx5UGhm?=
 =?utf-8?B?UitDSnBOR29SUWk3TVcvNnBtMk9yZjhYWWl0R3VjUElWWk9oVnlhRGlhVjVB?=
 =?utf-8?B?NU0rUnBCaDVqcXlUeThuMWNEbjRKV1Z0RzRySXBwU1VTeXFLRlFIN2VMZ1VC?=
 =?utf-8?B?VVhUaXBPWi9hSHJrZkZ3ZDgzelo1RkZwTHB0QldwaGxDZW44bTVXMmZMMFZ2?=
 =?utf-8?B?dGJrREkxNmZGN2xuYzdGWlU0QkZrKzNYYjMvMWpNS29tZ0RzR3lZVTh4dVBn?=
 =?utf-8?B?MmRxRm5QalY2VzZZaStQY1dpVjBuVUVPcWZaNWdZZUNiUnNNZHQ5cjM1ME9R?=
 =?utf-8?B?aXdYdzhkNDFFbEF0N0VDc3FIemtwVWlLV1h5V1U2Y0hRVlZwUkxZUU1iczIz?=
 =?utf-8?B?czV6S0JFby9xcjI1UTNUYlY5ZjdhN0VmT1pqanJ4Nm9HUHVBczh6cEk1WU93?=
 =?utf-8?B?czhDZjZqNThtK1QvempUV1hyVjRYR0hIaGV5bm1mSXRYYkJteXU3WkNjRXpD?=
 =?utf-8?B?aDF5WVlTN0JoZXVKWHRkbHhTbnlZTllsemxLdk1FN0ZPckRRVnk3MjJGczlI?=
 =?utf-8?B?bGlhSDQ0NDZhTFVyREJXUndBUHBNZmV2a1duZ2NoN3phZGFaM1hpZHhGY1Bq?=
 =?utf-8?B?Yis4RTFtcTBQUGM0YnpLeXNhbWt3YXZRc2N0TWdWcVF5dEJzdklmOTJxcHpZ?=
 =?utf-8?B?SU5RQ1BNQ0lYVk9qcG5JY3lHU1FKUkJZcmhpNlc2NklzVS9NbTBtR3JpU0R1?=
 =?utf-8?B?YTllNWJXNXdyOE15K0RYTm9ycFh0ekZqb1BLMlphOStKYlhJTUcyQkpuS0Nz?=
 =?utf-8?B?UWxvaHVGN3VLK1JvcGVuVWNYYkxoOU9EaWNoSFR0Vjg1RE5Qb1hGcUR5Vm1l?=
 =?utf-8?B?OFV5bWVMd0g4TWNLbWszWEVEbWllK0s5blNKUzREQThEdURKUmNROWpIMGNF?=
 =?utf-8?B?MjhTWGRpNFNxUXZnNUVLSzNvcmdlYzhQZ2pnQ3g3L3JheWw5eEFaaERTeXZF?=
 =?utf-8?B?aFNTVEJWbnJ0SXIyMlRxQXNCYUlJY1dsS1RScDNzZ2t1LzUveEhYYXJrR2FW?=
 =?utf-8?B?NldFODdvYmpoMVdJVVhoWmZqcEljVFEyZkhybWZvMjI1MHJWbkpXTW14MjdB?=
 =?utf-8?B?Q21laGorNi9PQUpnL2xvQzBHV0d2bjhUNDJkVUZoeHBqR2VnbDJiYTJPUy9z?=
 =?utf-8?B?eENLZk9ON3Bma1hDU0pSNlEzWEd4WHEySzZVUDRSQlhhNXI3NG5rcjd2aEI5?=
 =?utf-8?B?YUJWUkRqUW84Mlo4KzZrcU5HYUpxZE1hamk2QXA4aWlseC8xUFN0NkdzbHNh?=
 =?utf-8?Q?1c6j6tHUcUi9zlqbYZwrodIx8lebA4atEf5Rym6bBA=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 77e4481b-7dca-4f2c-fc17-08da03955429
X-MS-Exchange-CrossTenant-AuthSource: BN0PR11MB5744.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Mar 2022 19:28:32.6206
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 0Cyn+DYxeqfY62duuEKO2z0KiStUunaQ2RctdwURPxf1DHZOPYsTB0PHJBRxEd4HaZ8MT3pErKtqXtI6h6cu5spUw9ikkwWfM8NCtajvxT0=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB6008
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,RDNS_NONE,SPF_HELO_NONE,
	T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Jarkko,

On 3/11/2022 10:11 AM, Jarkko Sakkinen wrote:
> On Fri, Mar 11, 2022 at 09:53:29AM -0800, Reinette Chatre wrote:
>  
>> I do not believe that you encountered the #GP documented above because that
>> check is already present in the current implementation of
>> SGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS:
>>
>> sgx_ioc_enclave_restrict_permissions()->sgx_perm_from_user_secinfo():
>> 	if ((perm & SGX_SECINFO_W) && !(perm & SGX_SECINFO_R))
>> 		return -EINVAL;
>>
>> It does return EINVAL which is the catch-all error code used to represent
>> invalid input from user space. I am not convinced that EACCES should be used
>> instead though, EACCES means "Permission denied", which is not the case here.
>> The case here is just an invalid request.
>>
>> It currently does not prevent the user from setting PROT_NONE though, which
>> EMODPR does seem to allow.
>>
>> I saw Haitao's note that EMODPE requires "Read access permitted by enclave".
>> This motivates that EMODPR->PROT_NONE should not be allowed since it would
>> not be possible to relax permissions (run EMODPE) after that. Even so, I
>> also found in the SDM that EACCEPT has the note "Read access permitted
>> by enclave". That seems to indicate that EMODPR->PROT_NONE is not practical
>> from that perspective either since the enclave will not be able to
>> EACCEPT the change. Does that match your understanding?
>>
>> I will add the check for R in SGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS at least.
> 
> Yes, I think we are in the same line with this.
> 
> But there is another thing.
> 
> As EAUG is taken care by the page handler so should EMODPR. It makes the
> developer experience whole a lot easier when you don't have to back call
> to host and ask it to execute EMODPR for the range.
> 
> It's also a huge incosistency in this patch set that they are handled
> differently.
> 
> And it creates a concurrency case for user space that is complicated to say
> the least, i.e. divided work between host and enclave implementation to
> execute EMODPR is a nightmare scenario. On the other hand this is trivial
> to sort out in kernel.

EMODPR has possible failures due to state that is managed by the user space
runtime. Being able to communicate accurate EMODPR error codes to user space
runtime is helpful to the runtime in supporting its management of the enclave
memory. Accurate EMODPR error codes can be communicated when using an ioctl(),
not when run from within a page fault handler. 
 
> So what it means that, in one way or antoher, mprotect() needs to be the
> melting point for both.

mprotect() is the syscall to modify VMA permissions. EPCM permissions are
different from VMA permissions and they are currently treated differently
by the kernel. 

Moving EPCM permission changes to mprotect() forces EPCM permissions to be
the same as VMA permissions. That is a significant change. It is also
inconsistent since EPCM permission changes cannot be managed completely
from the kernel since the kernel can only ever restrict permissions.

> This can be called mandatory requirement, however
> this patch set it done, not least because of managing concurrency between
> kernel and user space.
> 
> You can get that done by these steps:
> 
> 1. Unmap PTE's in mprotect() flow.
> 2. In #PF handler, EMODPR with R set.

There is also the very significant ETRACK flow that
needs to be run after EMODPR. The implications of sending IPIs to all
CPUs that may be running in an enclave while in a page fault handler needs
to be considered. Page faults should be as fast as possible.

If this is considered then this tremendous impact on the page fault handler
should be managed and avoided as much as possible - but how will the page
fault handler even know when it should run EMODPR? The enclave can run 
EMODPE from within the enclave at any time without any insight from the
kernel so the only way to have accurate permissions would then be to
run EMODPR on _every_ page fault, which is obviously a non-starter due
to the significant impact (EMODPR and ETRACK) and blast radius (IPIs).

Trying to move running of EMODPR earlier, during the mprotect() call itself
is also full of obstacles since the mprotect() call may result in VMAs
being split, which is an operation that can fail, and followed by
the EMODPR-ETRACK flows that can also fail (and not be able to 
undo the VMA splits). With the EMODPR-ETRACK flows that can fail it
is here also not possible to communicate accurately to user space since
now there is the whole page range to consider, for example, mprotect()
cannot communicate
(a) which pages caused the failure, and (b) what failure was encountered.
This is possible when using the ioctl().


> This clear API for enclave developer because you know in what state pages
> are after mprotect(), and what you need to still do to them. Only the
> syscall needs to be them performed by the host side.

Supporting permission restriction in an ioctl() enables the runtime to manage
the enclave memory without needing to map it.

I have considered the idea of supporting the permission restriction with
mprotect() but as you can see in this response I did not find it to be
practical.

Reinette