Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Message-ID: <97565fed-dc67-bab1-28d4-c40201c9f055@intel.com>
Date:   Fri, 11 Mar 2022 09:53:29 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Firefox/91.0 Thunderbird/91.6.1
Subject: Re: [PATCH V2 16/32] x86/sgx: Support restricting of enclave page
 permissions
Content-Language: en-US
To:     Jarkko Sakkinen <jarkko@kernel.org>,
        Haitao Huang <haitao.huang@linux.intel.com>
CC:     "Dhanraj, Vijay" <vijay.dhanraj@intel.com>,
        "dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        "bp@alien8.de" <bp@alien8.de>,
        "Lutomirski, Andy" <luto@kernel.org>,
        "mingo@redhat.com" <mingo@redhat.com>,
        "linux-sgx@vger.kernel.org" <linux-sgx@vger.kernel.org>,
        "x86@kernel.org" <x86@kernel.org>,
        "Christopherson,, Sean" <seanjc@google.com>,
        "Huang, Kai" <kai.huang@intel.com>,
        "Zhang, Cathy" <cathy.zhang@intel.com>,
        "Xing, Cedric" <cedric.xing@intel.com>,
        "Huang, Haitao" <haitao.huang@intel.com>,
        "Shanahan, Mark" <mark.shanahan@intel.com>,
        "hpa@zytor.com" <hpa@zytor.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
References: <cover.1644274683.git.reinette.chatre@intel.com>
 <4ce06608b5351f65f4e6bc6fc87c88a71215a2e7.1644274683.git.reinette.chatre@intel.com>
 <YhLhoMFPyOFZ2fsX@iki.fi>
 <DM8PR11MB55917F499CDF4CC7D426B0A7F63C9@DM8PR11MB5591.namprd11.prod.outlook.com>
 <YimWaAqEnXHbLdjh@iki.fi> <op.1itu5vkewjvjmi@hhuan26-mobl1.mshome.net>
 <Yis8LV99mORcLYs6@iki.fi> <Yis9rA8uC/0bmWCF@iki.fi>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <Yis9rA8uC/0bmWCF@iki.fi>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Wm95VUZiRXZNQ2lUNlIxb0NJMXdxdnNsVVRDU2xsU3JyR1BBakV2RnFtVTZp?=
 =?utf-8?B?Ymt3SXFLOWFpNFJBckZ1cWsyMVdLeXNOa1V2U2ZSZ0NhMjF1WHVQQWVxdXlk?=
 =?utf-8?B?WkR3NnY2ckVqWjNLRStVU2F3VXB6c0NlaFhQU0w1eThzWlBvRDJZdmpOb3dk?=
 =?utf-8?B?NlNqdHZOUlV5TVY4dEhZZ1h0SXduWU9tOWF6OEduSFloWTUrMkdXVGFVTlRK?=
 =?utf-8?B?OWxiNDdsaEtnVi9jWkNRU1NYeUlIWGk0RzRSUFZsdEVSMDRka2dKRUJ6anIx?=
 =?utf-8?B?YU1CMlEzbDR4d1JGQUR2cThQUnpGWEM5djRld1Bib3BpdjJ2Y21NeTN5N2Jz?=
 =?utf-8?B?d0NqM3hOM1oxZ3RnUDVGWUZEZXVZM3VlR2dFVWhzTWJ5anR3WG55cDhaejha?=
 =?utf-8?B?MzhlUTdpdzVOMkN5bkZzTytNa3JHOFA2RnNjdnYyNGNVdVVDaWR3NGNpakdr?=
 =?utf-8?B?d0xHSmdXRFZpazNpeVlBbVRDU2t2WFVRa2RRVFRlZHRZQlNIZjJQcFVRU0ZI?=
 =?utf-8?B?cjNMa015TDdqWXEybzcwdXR0bkZPWGhZVkFQQVE0dmsvczFhckhqaHBiNG8r?=
 =?utf-8?B?WWltTTJYS2xQN0pldU0rMFk4Qi92VGNnMEVxRnpUTHg2N3ZSTkNDTk9kcUo5?=
 =?utf-8?B?cG43MDhuWnlHeHlmUWRjTitNb2toQWhVMUY3VU1hMytpR0IyaWUwZEZZZzVO?=
 =?utf-8?B?eE0zMWJoekVHV1hPOEF3TmwxbitOYlU1Vk04eFl6YVB5SnhqQU5oclR4L1Zx?=
 =?utf-8?B?L3M1Z05ZcXJHSWlDa2s5UEpSUmppTExQUENiemNha1FDK3lQQ0hrS1dNcmM3?=
 =?utf-8?B?NTVjcVBYZ0JUZGswNlE4UXpFM096QUZScTVLTTB4M0FvU1MzSkcrNmRsWHFy?=
 =?utf-8?B?eWNNYm14ZzNTNk9xN0xDRHFiVU5NbmQyV2M2UDBUTzROUTZsYTcySnNQOFo4?=
 =?utf-8?B?d1B5NmxRcjBTaTQ2Sy9Ed2pLeVBWRUZLbWxQbVA1Nmh5Q0dNWUFrYmxBM1dJ?=
 =?utf-8?B?WDRINlNjS1hLTHlpUklwQVBrSEcxbnllVysxTDZSVTJBT3lhUGJqR2N1Njlm?=
 =?utf-8?B?SmNxMlBxV3dwR1poMFhRc0M4SEdWYVlOVC9SQ0FLWWE4bzBjL21LM0hIclhD?=
 =?utf-8?B?TzRVWEtQRnB6TFVhNWRnWmh2SXg2a0g5UTMyOFVvMkFUbnNJRmVpVStGRk1F?=
 =?utf-8?B?ekpQcHhrYWlGSWhOcG9lZURkZmZmMFBPYm5JcFFvUkFReURQamFQazlVYjFq?=
 =?utf-8?B?SUtxOHRFZG5PRDZEMjMyTkFFaFBOU0MzcEJaSzM4cjhIY1hSM0M5MmlFVmtN?=
 =?utf-8?B?NStvc2pLOEpXZXkzNnpmN2tqSTNWVlpSeks3SnZ4eGpLVVRRQnNTdXhUNWQ3?=
 =?utf-8?B?Tkt1Mkl2NlU3TkI1eWZONDhmckw4ZW85VVd5Q3JSUkhiQ1RUM08zWGR0V09l?=
 =?utf-8?B?eUdVdUltQUt6dkFKNERCejAzMEhzV091RmhOME1tYnRBQjk0c1pqNXN0UHhO?=
 =?utf-8?B?c0tZUHRtL2QxT1VlRUlEQnRRNnJ6RHJnRVlPbXZFZG9OU0RWcU5udkQ5azl5?=
 =?utf-8?B?ZzVlaHU1bDdHb2xCbldvb2s4dXZqTDJUR2IvblJuK1FCQjFhTlkvdDJFYXc0?=
 =?utf-8?B?NzZqY0pmSEtJMGZDdjFRM1hmU2xwUmFuVzJ0QU1acUU3eGlibWNjT2ptNitI?=
 =?utf-8?B?UnNETzlyTDVTNFNuUEk3a2MyaTJFTEx2cDdTQW5penhnRHRzUEkyY0JUZ1M1?=
 =?utf-8?B?ck51TjhOK3VhYnlTaVlRZXpCL1Z1SHRBcW1IbUxEQlZmb2ZuaU5XeGRFb0xK?=
 =?utf-8?B?QTBRWnNQaXlMdmpjcFR0R1A0QVVqNUc4SVpJU1VYMWxzK3hrUjJUdWNqOU1u?=
 =?utf-8?B?Y2UzRzdOQ1NKeXdjUjBlZmp2ajkvV2N2YTI5T25FKzRyMTZ0eHQ0cGdsTFEr?=
 =?utf-8?B?L2xOUjhUZWUzcXRVdEpzZjcwS3J0ajBuQ2FPNmEyczNwSlFPZnlxbnVTeTFV?=
 =?utf-8?B?MkdCdksrU1RIMUh4ZkhhZHcrQkJSSVZHRjFMSkFWM0pYem1Fa3owN1pjVEc1?=
 =?utf-8?B?b2ZTekhYMjAvampRUFRtQjByM3RlY1hVUnV5UXluM1lscTNsM3BubUF1TU1w?=
 =?utf-8?B?RFdwWkFBN0hhbm93U0lUakJPTS91VURlUFhvdmI1Y3JpNHp4d2wrVTZvNHlx?=
 =?utf-8?B?Rnk3Wm9mZHJDQjRYWVlJMXpyOGhFQVRzZXRFeGlJbmN0NHUrTnlzSTB3Q01K?=
 =?utf-8?B?N1dYVkxlSHZOcytwM2FSQTFDbDJRPT0=?=
X-MS-Exchange-CrossTenant-Network-Message-Id: fa7118fd-8f31-4d55-9e47-08da03881015
X-MS-Exchange-CrossTenant-AuthSource: BN0PR11MB5744.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Mar 2022 17:53:35.0071
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: FaD0kfr/xSO3W91G46gv6Q+ra1tTCh4KQtrkFZ/wpdkrBmTbcQp6O69dC8LqBrgPgv2RnYPpkz4ElTzV4dQVMHBVakpQXT8Wbh2DZPfsvn4=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR11MB2658
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,RDNS_NONE,SPF_HELO_NONE,
	T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Jarkko,

On 3/11/2022 4:16 AM, Jarkko Sakkinen wrote:
> On Fri, Mar 11, 2022 at 02:10:24PM +0200, Jarkko Sakkinen wrote:
>> On Thu, Mar 10, 2022 at 12:33:20PM -0600, Haitao Huang wrote:
>>> Hi Jarkko
>>>
>>> I have some trouble understanding the sequences below.
>>>
>>> On Thu, 10 Mar 2022 00:10:48 -0600, Jarkko Sakkinen <jarkko@kernel.org>
>>> wrote:
>>>
>>>> On Wed, Feb 23, 2022 at 07:21:50PM +0000, Dhanraj, Vijay wrote:
>>>>> Hi All,
>>>>>
>>>>> Regarding the recent update of splitting the page permissions change
>>>>> request into two IOCTLS (RELAX and RESTRICT), can we combine them into
>>>>> one? That is, revert to how it was done in the v1 version?
>>>>>
>>>>> Why? Currently in Gramine (a library OS for unmodified applications,
>>>>> https://gramineproject.io/) with the new proposed change, one needs to
>>>>> store the page permission for each page or range of pages. And for every
>>>>> request of `mmap` or `mprotect`, Gramine would have to do a lookup
>>>>> of the
>>>>> page permissions for the request range and then call the respective
>>>>> IOCTL
>>>>> either RESTRICT or RELAX. This seems a little overwhelming.
>>>>>
>>>>> Request: Instead, can we do `MODPE`,  call `RESTRICT` IOCTL, and then do
>>>>> an `EACCEPT` irrespective of RELAX or RESTRICT page permission request?
>>>>> With this approach, we can avoid storing  page permissions and simplify
>>>>> the implementation.
>>>>>
>>>>> I understand RESTRICT IOCTL would do a `MODPR` and trigger `ETRACK`
>>>>> flows
>>>>> to do TLB shootdowns which might not be needed for RELAX IOCTL but I am
>>>>> not sure what will be the performance impact. Is there any data point to
>>>>> see the performance impact?
>>>>>
>>>>> Thanks,
>>>>> -Vijay
>>>>
>>>> This should get better in the next versuin. "relax" is gone. And for
>>>> dynamic EAUG'd pages only VMA and EPCM permissions matter, i.e.
>>>> internal vm_max_prot_bits is set to RWX.
>>>>
>>>> I patched the existing series eno
>>>>
>>>> For Enarx I'm using the following patterns.
>>>>
>>>> Shim mmap() handler:
>>>> 1. Ask host for mmap() syscall.
>>>> 2. Construct secinfo matching the protection bits.
>>>> 3. For each page in the address range: EACCEPTCOPY with a
>>>>    zero page.
>>>
>>> For EACCEPTCOPY to work, I believe PTE.RW is required for the target page.
>>> So this only works for mmap(..., RW) or mmap(...,RWX).
>>
>> I use it only with EAUG.
>>
>>> So that gives you pages with RW/RWX.
>>>
>>> To change permissions of any of those pages from RW/RWX to R/RX , you need
>>> call ENCLAVE_RESTRICT_PERMISSIONS ioctl with R or with PROT_NONE. you can't
>>> just do EMODPE.
>>>
>>> so for RW->R, you either:
>>>
>>> 1)EMODPR(EPCM.NONE)
>>> 2)EACCEPT(EPCM.NONE)
>>> 3)EMODPE(R) -- not sure this would work as spec says EMODPE requires "Read
>>> access permitted by enclave"
>>>
>>> or:
>>>
>>> 1)EMODPR(EPCM.PROT_R)
>>> 2)EACCEPT(EPCM.PROT_R)
>>
>> I checked from SDM and you're correct.
>>
>> Then the appropriate thing is to reset to R.
>>
>>>> Shim mprotect() handler:
>>>> 1. Ask host for mprotect() syscall.
>>>> 2. For each page in the address range: EACCEPT with PROT_NONE
>>>>    secinfo and EMODPE with the secinfo having the prot bits.
>>>
>>> EACCEPT requires PTE.R. And EAUG'd pages will always initialized with
>>> EPCM.RW,
>>> so EACCEPT(EPCM.PROT_NONE) will fail with SGX_PAGE_ATTRIBUTES_MISMATCH.
>>
>> Ditto.
>>
>>>> Backend mprotect() handler:
>>>> 1. Invoke ENCLAVE_RESTRICT_PERMISSIONS ioctl for the address
>>>>    range with PROT_NONE.
>>>> 2. Invoke real mprotect() syscall.
>>>>
>>> Note #1 can only be done after EACCEPT. MODPR is not allowed for pending
>>> pages.
>>
>> Yes, and that's what I'm doing. After that shim does EACCEPT's in a loop.
>>
>> Reinette, the ioctl should already check that either R or W is set in
>> secinfo and return -EACCES.
>>
>> I.e.
>>
>> (* Check for misconfigured SECINFO flags*)
>> IF ( (SCRATCH_SECINFO reserved fields are not zero ) or
>> (SCRATCH_SECINFO.FLAGS.R is 0 and SCRATCH_SECINFO.FLAGS.W is not 0) )
>> THEN #GP(0); FI;
>>
>> I was testing this and wondering why my enclave #GP's, and then I checked
>> SDM after reading Haitao's response. So clearly check in kernel side is
>> needed.

I do not believe that you encountered the #GP documented above because that
check is already present in the current implementation of
SGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS:

sgx_ioc_enclave_restrict_permissions()->sgx_perm_from_user_secinfo():
	if ((perm & SGX_SECINFO_W) && !(perm & SGX_SECINFO_R))
		return -EINVAL;

It does return EINVAL which is the catch-all error code used to represent
invalid input from user space. I am not convinced that EACCES should be used
instead though, EACCES means "Permission denied", which is not the case here.
The case here is just an invalid request.

It currently does not prevent the user from setting PROT_NONE though, which
EMODPR does seem to allow.

I saw Haitao's note that EMODPE requires "Read access permitted by enclave".
This motivates that EMODPR->PROT_NONE should not be allowed since it would
not be possible to relax permissions (run EMODPE) after that. Even so, I
also found in the SDM that EACCEPT has the note "Read access permitted
by enclave". That seems to indicate that EMODPR->PROT_NONE is not practical
from that perspective either since the enclave will not be able to
EACCEPT the change. Does that match your understanding?

I will add the check for R in SGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS at least.

> I would consider also adding such check "add pages". It's our least common
> denominator.
> 
> If we can assume that at least R is there for every enclave page, then it
> gives invariant that enables EMODPR with R all the time.

Adding pages without permissions to an enclave does not seem practical. I
do not know if there are such usages. I can add this as a separate change for
consideration.

Reinette