Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <acc54cf1-09d8-3ceb-be20-e9dc553259df@intel.com>
Date:   Wed, 9 Mar 2022 16:11:13 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Firefox/91.0 Thunderbird/91.6.1
Subject: Re: [RFC PATCH v2.1 14/30] x86/sgx: Support restricting of enclave
 page permissions
Content-Language: en-US
To:     Jarkko Sakkinen <jarkko@kernel.org>
CC:     <linux-sgx@vger.kernel.org>,
        Nathaniel McCallum <nathaniel@profian.com>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        "maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT)" <x86@kernel.org>,
        "H. Peter Anvin" <hpa@zytor.com>,
        "open list:X86 ARCHITECTURE (32-BIT AND 64-BIT)" 
        <linux-kernel@vger.kernel.org>
References: <20220304093524.397485-1-jarkko@kernel.org>
 <20220304093524.397485-14-jarkko@kernel.org> <YihqwiU3Dr5mvMQx@iki.fi>
 <Yih00gFMaJhLIgxa@iki.fi> <5e89a3be-0760-b1b4-7693-2f3d9ac5066b@intel.com>
 <Yik51tjxXcqamBrO@iki.fi> <Yik7cdUif35yA7ui@iki.fi>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <Yik7cdUif35yA7ui@iki.fi>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?LzFMYkU2Z3owdTZWWTBKVCtFNFd4ODBqMysrN3FtbUVOZUVBa2hoV2pUM3NG?=
 =?utf-8?B?NjNYSWhiSDFUclJWdFFoak1DeDRJQ0tmRnc2Zk5xUWZNNVFQSHNGSlJoUERR?=
 =?utf-8?B?NmMwVTFvdFpLekRUNzU1Z094aFB2RzgvRlQ1cWhOVi9VaU4zMGJMVEovREhw?=
 =?utf-8?B?QkJTeHQ1UGR0dTROd1BrZnNzeDBkTlo5TXRuWW5LKzFrZkRKTWR0ZWpqYjVv?=
 =?utf-8?B?bDIyTE9sVFkzNzBHL3FTalRvc29pOElMb2dKT0dqai9PWWFJTGtMR2hoczlU?=
 =?utf-8?B?MnFkQkVHNGowRWhpNVlLM3FXZGhYS0Eyb09WOXFMRmJIOTdCWmM5VnFDVVQ5?=
 =?utf-8?B?QTlCL0QvWFkxeU1XTW4vdHV1V2pMSWVJcXpreklzTGtvQ0F4L3pHL3FRbGc1?=
 =?utf-8?B?TVpkL0RrOUVIZS9VRDAzWXFhNTJ0UGN6cHhacVVCUHNPOU12UjVLdGxuSVkw?=
 =?utf-8?B?QjZrNmJReWZmRFRRU3FDU1Z5NG1jT2sxNDBzR2I3WVpqT3EzaVZOOThtd0FQ?=
 =?utf-8?B?aFY2MkszaTJuNEU5WHJpb3krbzhmY3R0eGZRMDFaaCswczBxOUxIc0Z4amNK?=
 =?utf-8?B?WXF3RDk2QUJFeEUramFUWmV2S2NJL1FFblJjaThYTkVWUlFuRStjVFdodzRG?=
 =?utf-8?B?Y0hxczYzd2R2eXJEVWNtL1lWaU4xam9Mb1lEc1dTQVZVUHgyWXpSbTRpWVFq?=
 =?utf-8?B?aVBaMjBJbVY5RmJWREdFaC90M3VleHc4UDUvZUxKSDE2SC9rUXdpeng2NUlx?=
 =?utf-8?B?Y1p2Qk1BZ0I0WXAycVJydzQrdFRDem1ocCtIOUJxd3BVY0lZL0d0UHRrRHNj?=
 =?utf-8?B?a1dlb1drVnNwQ014YmNHRDRVOStMa1ByeHl0ajFXU1d0dEJCNkFwS2M0TmI3?=
 =?utf-8?B?TFdxb3BvUFRacXJlcm9Ibjdsdmc0L1JPQjJ6OUVuT2xIZUhQNExjYk1iNUVq?=
 =?utf-8?B?cjY5ZkNwZUMwbldxMVNtU2ZNRDJ0a2RrYUs0M3l3aHk3ZHIzUklqUWdueU5i?=
 =?utf-8?B?VXBraHZZUWh5V1RCU3FmZkJrMmo5SDh0UzRRNkNZbk16bll4RVk5U01qajFs?=
 =?utf-8?B?blRhcWtBVGxYR0luV1FhTThOTGpMbWx2SmxPRExXcjBzWEtqNXZPSWUwZ1A1?=
 =?utf-8?B?cC9ubjlnbjRsalo0cERESmd4RXJUSURqaVl1S2o4K29VM2c4VENYRlZXcHU5?=
 =?utf-8?B?eVdlNkVvdFErUTJJakVET0ZBQnp2NHFCOTUyRFc2RVhhZDRyejFsWm5wRTlT?=
 =?utf-8?B?OENsM1dzRllMUlRCd0JyRmZxUzBqS2MyLzdnMlZMa0FrZ3ltSTd6OThNTTl6?=
 =?utf-8?B?UDcxMitFdC9KMnorSG5wVm9Qb09IZjFmWUVWc3FLZStrUU1xUTk4MFdDdmhN?=
 =?utf-8?B?OXBZVUx1QURtOXhRQkxCNytibEVyelN2dHA2UlZnbHF1OU5qQnVSeGdpK056?=
 =?utf-8?B?a0pJTnNkZkU3T0I5Z2dZbnVBVUVPUWNNY1JrUFJIMVFkQ0hTdS9BaUpPSmwy?=
 =?utf-8?B?dDN1OVhEMkZvK2U2UER6UjQySkJ1dmgyc3daTWtIZmZmUFZqejR3ZFp1R2Jk?=
 =?utf-8?B?SUVQeE1aTGJ6ZmNiUDgzUlZpUGFWaHo1a2FhVytDdlluc0N1Y2oxVVB0Qmdp?=
 =?utf-8?B?bjBMYXpoMmIzbTZpa1lJVEszamVzUHM5dHVheDVSSk1NTEF5OVBPT2krL2Y4?=
 =?utf-8?B?SmxQR2haM2F4eXlyS29NdlhBYkUrR2FyeVh3bUZnWTRldVhscFFXRHhNQ2gz?=
 =?utf-8?B?YURPNWZwT3NYaTdXYU52NmppTzJZcHV4Qnc2TkNXbHlsYWRVUlF2UkhxaHR2?=
 =?utf-8?B?MXoxRGlVL0lPQS9FY256S252aEIzTGpuTGlIR3hLcHA3TnJ0c1dDWERsWkZB?=
 =?utf-8?B?MEZpK1g4RlhsQlljR0NpT1BiNW9qSCtieEdmRTkyM2R0YUJ3UG1GSEFaVENT?=
 =?utf-8?B?d1dXMzJiaDN6dm41eWsyVjdERllXOGt0OEJYOEtVZUVaWDZQVS8yTCtxRmhi?=
 =?utf-8?B?b2tiOFJROW1BPT0=?=
X-MS-Exchange-CrossTenant-Network-Message-Id: ca2202a3-50f8-43d2-fe4a-08da022a7f17
X-MS-Exchange-CrossTenant-AuthSource: BN0PR11MB5744.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Mar 2022 00:11:17.2225
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: PezBFZ7eNEYsESSB9OCd9Co3b2Rt36b/H6SKZMJJuUr4+64m0EnduRTsduN9yoMjUyugTDOlZoji1emsVEB+Y2jz6NS63QKIDa4lZBRD3ew=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR11MB5315
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.9 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Jarkko,

On 3/9/2022 3:42 PM, Jarkko Sakkinen wrote:
> On Thu, Mar 10, 2022 at 01:35:53AM +0200, Jarkko Sakkinen wrote:
>> On Wed, Mar 09, 2022 at 08:59:42AM -0800, Reinette Chatre wrote:
>>> Hi Jarkko,
>>>
>>> On 3/9/2022 1:35 AM, Jarkko Sakkinen wrote:
>>>> On Wed, Mar 09, 2022 at 10:52:22AM +0200, Jarkko Sakkinen wrote:
>>>>> On Fri, Mar 04, 2022 at 11:35:08AM +0200, Jarkko Sakkinen wrote:
>>>>>> +#define SGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS \
>>>>>> +	_IOWR(SGX_MAGIC, 0x05, struct sgx_enclave_restrict_perm)
>>>>>
>>>>> What if this was replaced with just SGX_IOC_ENCLAVE_RESET_PAGES, which
>>>>> would simply do EMODPR with PROT_NONE? The main ingredient of EMODPR is to
>>>>> flush out the TLB's, and move a page to pending state, which cannot be done
>>>>> from inside the enclave.
>>>
>>> I see the main ingredient as running EMODPR to restrict the EPCM permissions. If
>>> the user wants to use SGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS just to flush TLB it is
>>> already possible since attempting to use EMODPR to relax permissions does not
>>> change any permissions (although it still sets EPCM.PR) but yet will still
>>> flush the TLB.
>>
>> It's not just to flush the TLB. It also resets permissions to zero from
>> which it is easy to set the exact permissions with EMODPE.
>>
>>> Even so, you have a very good point that removing SGX_IOC_ENCLAVE_RELAX_PERMISSIONS
>>> removes the ability for users to flush the TLB after an EMODPE. If there are
>>> thus PTEs present at the time the user runs EMODPE the pages would not be
>>> accessible with the new permissions.
>>>
>>> Repurposing SGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS with PROT_NONE to accomplish
>>> this is not efficient because:
>>> - For the OS to flush the TLB the enclave pages need not be in the EPC but
>>>   in order to run EMODPR the enclave page needs to be in the EPC. In an 
>>>   oversubscribed environment running EMODPR unnecessarily can thus introduce
>>>   a significant delay. Please see the performance comparison I did in
>>>   https://lore.kernel.org/linux-sgx/77e81306-6b03-4b09-2df2-48e09e2e79d5@intel.com/
>>>   The test shows that running EMODPR unnecessarily can be orders of magnitude slower.
>>> - Running EMODPR on an enclave page sets the EPCM.PR bin in the enclave page
>>>   that needs to be cleared with an EACCEPT from within the enclave.
>>>   If the user just wants to reset the TLB after running EMODPE then it should
>>>   not be necessary to run EACCEPT again to reset EPCM.PR.
>>>
>>> Resetting the TLB is exactly what SGX_IOC_ENCLAVE_RELAX_PERMISSIONS did in an 
>>> efficient way - it is quick (no need to load pages into EPC) and it does not
>>> require EACCEPT to clear EPCM.PR. 
>>>
>>> It looks like we need SGX_IOC_ENCLAVE_RELAX_PERMISSIONS back. We could
>>> rename it to SGX_IOC_ENCLAVE_RESET_PAGES if you prefer.
>>
>> Please do not add it. We do not have any use for it. It's not only used
>> to flush TLB's so it would not do any good. I just use it with fixed
>> PROT_NONE permissions.
>>
>>>>> It's there because of microarchitecture constraints, and less so to work as
>>>>> a reasonable permission control mechanism (actually it does terrible job on
>>>>> that side and only confuses).
>>>>>
>>>>> Once you have this magic TLB reset button in place you can just do one
>>>>> EACCEPT and EMODPE inside the enclave and you're done.
>>>>>
>>>>> This is also kind of atomic in the sense that EACCEPT free's a page with no
>>>>> rights so no misuse can happend before EMODPE has tuned EPCM.
>>>>
>>>> I wonder if this type of pattern could be made work out for Graphene:
>>>>
>>>> 1. SGX_IOC_ENCLAVE_RESET_PAGES
>>>> 2. EACCEPT + EMODPE
>>>>
>>>> This kind of delivers EMODP that everyone has been looking for.
>>>
>>> EACCEPT will result in page table entries created for the enclave page. EMODPE
>>> will be able to relax the permissions but TLB flush would be required to
>>> access the page with the new permissions. SGX_IOC_ENCLAVE_RELAX_PERMISSIONS
>>> (renamed to SGX_IOC_ENCLAVE_RESET_PAGES?) that does just a TLB flush is
>>> required to be after EMODPE.
>>
>> For EMODPE TLB flush is not required. I even verified this from Mark
>> Shanahan. And since access rights are zero, the page cannot be
>> deferenced by threads before EMODPE.
> 
> I'm fine of course keeping SGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS as it
> is too. It's at least future-proof that way.

Will do.

Reinette