Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <690d3d8e-6214-dcdd-daaa-48a380114ad7@intel.com>
Date:   Thu, 17 Mar 2022 15:08:04 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Firefox/91.0 Thunderbird/91.7.0
Subject: Re: [PATCH V2 16/32] x86/sgx: Support restricting of enclave page
 permissions
Content-Language: en-US
To:     Jarkko Sakkinen <jarkko@kernel.org>
CC:     Haitao Huang <haitao.huang@linux.intel.com>,
        "Dhanraj, Vijay" <vijay.dhanraj@intel.com>,
        "dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        "bp@alien8.de" <bp@alien8.de>,
        "Lutomirski, Andy" <luto@kernel.org>,
        "mingo@redhat.com" <mingo@redhat.com>,
        "linux-sgx@vger.kernel.org" <linux-sgx@vger.kernel.org>,
        "x86@kernel.org" <x86@kernel.org>,
        "Christopherson,, Sean" <seanjc@google.com>,
        "Huang, Kai" <kai.huang@intel.com>,
        "Zhang, Cathy" <cathy.zhang@intel.com>,
        "Xing, Cedric" <cedric.xing@intel.com>,
        "Huang, Haitao" <haitao.huang@intel.com>,
        "Shanahan, Mark" <mark.shanahan@intel.com>,
        "hpa@zytor.com" <hpa@zytor.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
References: <DM8PR11MB55917F499CDF4CC7D426B0A7F63C9@DM8PR11MB5591.namprd11.prod.outlook.com>
 <YimWaAqEnXHbLdjh@iki.fi> <op.1itu5vkewjvjmi@hhuan26-mobl1.mshome.net>
 <Yis8LV99mORcLYs6@iki.fi> <Yis9rA8uC/0bmWCF@iki.fi>
 <97565fed-dc67-bab1-28d4-c40201c9f055@intel.com> <YiuQx+X9UQ2l22un@iki.fi>
 <e1c04077-0165-c5ec-53be-7fd732965e80@intel.com> <Yi65sM+yCvZU0/am@iki.fi>
 <7ff5e217-4042-764b-3d32-49314f00ff54@intel.com> <YjK5ZWJRQX+lyUxS@iki.fi>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <YjK5ZWJRQX+lyUxS@iki.fi>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dG55MkRNUjN3cUVRM1N5V2dHRWVqZllwaDRiVVRQMm5SbDlFOExRV21FVEtn?=
 =?utf-8?B?NGVFSytXWE9naEVlZXhUK25xek1vcmdnTFhIenhDd2l6R3Z2R0VhQk9UcHVq?=
 =?utf-8?B?amlMM0hRVG52VUhDanI2UFhKVUZkL2tLS09xWTBBV1Y0aXBIZG1nUjVjVm5l?=
 =?utf-8?B?NW41MlpqNm5zWiszeiswWmlLYzVUZ1FETkZLempkMkgwdzZyVk9TSXFETi9O?=
 =?utf-8?B?bDJvZFhBalZUQkozSUwzTkowV0JoSWc0b1YwT2RoQkxzV2d0REZEUmxRRjc1?=
 =?utf-8?B?Nkd4elJaZUVzM2RPRVdDZWMveFVFYVQ3NFZFTUhuZ0Jua3FkZUZHVlRmYWY3?=
 =?utf-8?B?T0xsdE84QUNrOXEvTVlBTXpWQjVFL0x1WHcwSXBIMzF1Z2QwY1RaRnFZSklD?=
 =?utf-8?B?cWt4TEs0bEwzOUxIOEt4RGljVkZYSzNxbDljc05PMy92d1h6MkVTd1I3K043?=
 =?utf-8?B?K2taVHV0OU1ONm1GcHJ3ZytCa0JzY2JZSzByK0dYRll5ZGJLN0ZhNjRYekpj?=
 =?utf-8?B?VlFiSWVwcCtlOG9SdDF5bmJ3V0haQmRLS3hkQ1dkYllNMUJrVGNtN09FZnVz?=
 =?utf-8?B?N0ZXUjI5NTlUVEVNNzBXNmtJaUVNU1JWRHpVZkQxMVdZbmVQamlUK3ZKUFhn?=
 =?utf-8?B?dFdRUTdFL3BkdngvbVZjTHFWWmFrWjcwUjhTdTNoS0lTSmdleWxpZG03V2VM?=
 =?utf-8?B?SWhZdVorSXVMTktQcVZZdStOSHNhd3RKaHVxSTVHOFdvdXlQR08rN3Y1emNq?=
 =?utf-8?B?MG1SQ2Y2d3FWeUI5eVRFbkROTHRDNS9lcitxNEhIdk41aEh6L2pnU01YSTRs?=
 =?utf-8?B?U2sxQVN4V21ldDBXNmFDY2Vjcmdtd2p6cVhjYVRGZEF2RU1IQUR1V3VhSW9L?=
 =?utf-8?B?WlZrMVk5WENsb25pRUhPTEtOMzNhNnoyR25aZ0tWUDhmc0xVVlhNV3UrY3RV?=
 =?utf-8?B?dnhZNXNFUXBpejFGRU91aHdReStQdlRiWjZLK0lXQk8zbGlneWpzK214WEhT?=
 =?utf-8?B?SkxlamsrYVQ4bXZ1S3ZVRUEzcWh5QkNENk9XalBCN3kvZDc3dDh4RlJTd0Rx?=
 =?utf-8?B?QjVQaWNGVURRZFZtS1JWQ2ppbm1EcUw5b21nQVFJWXhIY1lFcjJKTFFoOTlK?=
 =?utf-8?B?WGp4emZ5N3ZUUWs2T3JpcEFlUml1WWcySFlmM2RpZzZyR05rT1dLd3BUOTdL?=
 =?utf-8?B?KytQN1dlVFJIQzY2NFEyc2pPYVNoekNwQ0xvdkQxQkRwYzhRQngzYjJkUEFj?=
 =?utf-8?B?QU51NTRsSnBRc0l1dFdtb044T3ppMmI0cHkrb0xCSk9ydlBZd0ZHT1lobXpt?=
 =?utf-8?B?cGoyQzZuRktvR0JRdmJsOEIwc01NZHVQUkhGTmR5M3ZQU1VDbWFjZXBRczVw?=
 =?utf-8?B?OGc3a29mNFZjNzZEc1pydjFzU1VscXBEdjkrUW12UkhDZUpKdmYySkNTQXBC?=
 =?utf-8?B?YUdFVHpZSlpZTGxLY25LYlJSRnEwK3JTRU9LSmNhQ2RCSDFKaFVrTjRZZTY3?=
 =?utf-8?B?SHhRRW9DUWNHVVpnTTdWZU5yYUtwK1RRbkxySFArUWlYcXRRc0p1c1VwR01C?=
 =?utf-8?B?bS9ISWcxcG85MUs1YVBCYzV6RC9jb3lvSlVWaVVlTEh0citLbWlDOEFKc2hU?=
 =?utf-8?B?bTBpd1hRdW11YXdBdS8wV0NydlFzWDJ6UDVnSDUwbkxveEdNT1k5UmpESjVS?=
 =?utf-8?B?SklPL1ZCT0tNRFNJV0QxUWdZS1gxdzdWekZHMDk5YkZicW1wSTVMemM4Yko0?=
 =?utf-8?B?Qm9NZ0JYa2o1c1ltUEFYTnJDWjZNdlBiZnhaKzZaVCtVU0FZVHpOcjFnWVVv?=
 =?utf-8?B?NUI3clNkSDFrKzlxZHd2TlF1ODgxVE44VnJNaVNzRkthbVd4K0pUSFBBSisz?=
 =?utf-8?B?MDR2WFhUY3U3dzNWdTRxRjdSejhKeVArQm5MaUsvUk5aazdyY0FsOGh6VXRF?=
 =?utf-8?B?eEdFaFVybGxJZ3BhWDJEaU53OS9FYmIrYXliRnkyR2dkVU9yZXdKNmg0cmRX?=
 =?utf-8?B?REMwZ05aNEFBPT0=?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 73d017fb-0bb5-4390-fac9-08da08629f22
X-MS-Exchange-CrossTenant-AuthSource: BN0PR11MB5744.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Mar 2022 22:08:09.9150
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: rvAtzBcdmr6tS4FZa5i2PFBEG9TZ3bFj/kDqWi1FjOKU6jmbXJYqk4aI5nJ7mDCC2aj4WyV5rDMnzYQy5OfTCAiQcvrVR+19QlFbHHQPAbk=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB4628
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_LOW,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Jarkko,

On 3/16/2022 9:30 PM, Jarkko Sakkinen wrote:
> On Mon, Mar 14, 2022 at 08:32:28AM -0700, Reinette Chatre wrote:
>> Hi Jarkko,
>>
>> On 3/13/2022 8:42 PM, Jarkko Sakkinen wrote:
>>> On Fri, Mar 11, 2022 at 11:28:27AM -0800, Reinette Chatre wrote:
>>>> Supporting permission restriction in an ioctl() enables the runtime to manage
>>>> the enclave memory without needing to map it.
>>>
>>> Which is opposite what you do in EAUG. You can also augment pages without
>>> needing the map them. Sure you get that capability, but it is quite useless
>>> in practice.
>>>
>>>> I have considered the idea of supporting the permission restriction with
>>>> mprotect() but as you can see in this response I did not find it to be
>>>> practical.
>>>
>>> Where is it practical? What is your application? How is it practical to
>>> delegate the concurrency management of a split mprotect() to user space?
>>> How do we get rid off a useless up-call to the host?
>>>
>>
>> The email you responded to contained many obstacles against using mprotect()
>> but you chose to ignore them and snipped them all from your response. Could
>> you please address the issues instead of dismissing them? 
> 
> I did read the whole email but did not see anything that would make a case
> for fully exposed EMODPR, or having asymmetrical towards how EAUG works.

I believe that on its own each obstacle I shared with you is significant enough
to not follow that approach. You simply respond that I am just not making a
case without acknowledging any obstacle or providing a reason why the obstacles
are not valid.

To help me understand your view, could you please respond to each of the
obstacles I list below and how it is not an issue?


1) ABI change:
   mprotect() is currently supported to modify VMA permissions
   irrespective of EPCM permissions. Supporting EPCM permission
   changes with mprotect() would change this behavior.
   For example, currently it is possible to have RW enclave
   memory and support multiple tasks accessing the memory. Two
   tasks can map the memory RW and later one can run mprotect()
   to reduce the VMA permissions to read-only without impacting
   the access of the other task.
   By moving EPCM permission changes to mprotect() this usage
   will no longer be supported and current behavior will change.
   
2) Only half EPCM permission management:
   Moving to mprotect() as a way to set EPCM permissions is
   not a clear interface for EPCM permission management because
   the kernel can only restrict permissions. Even so, the kernel
   has no insight into the current EPCM permissions and thus whether they
   actually need to be restricted so every mprotect() call,
   all except RWX, will need to be treated as a permission
   restriction with all the implementation obstacles
   that accompany it (more below).

There are two possible ways to implement permission restriction
as triggered by mprotect(), (a) during the mprotect() call or
(b) during a subsequent #PF (as suggested by you), each has
its own obstacles.

3) mprotect() implementation 

   When the user calls mprotect() the expectation is that the
   call will either succeed or fail. If the call fails the user
   expects the system to be unchanged. This is not possible if
   permission restriction is done as part of mprotect().

   (a) mprotect() may span multiple VMAs and involves VMA splits
       that (from what I understand) cannot be undone. SGX memory
       does not support VMA merges. If any SGX function
       (EMODPR or ETRACK on any page) done after a VMA split fails
       then the user will be left with fragmented memory.

   (b) The EMODPR/ETRACK pair can fail on any of the pages provided
       by the mprotect() call. If there is a failure then the
       kernel cannot undo previously executed EMODPR since the kernel
       cannot run EMODPE. The EPCM permissions are thus left in inconsistent
       state since some of the pages would have changed EPCM permissions
       and mprotect() does not have mechanism to communicate
       partial success.
       The partial success is needed to communicate to user space
       (i) which pages need EACCEPT, (ii) which pages need to be
       in new request (although user space does not have information
       to help the new request succeed - see below).

   (c) User space runtime has control over management of EPC memory
       and accurate failure information would help it to do so.
       Knowing the error code of the EMODPR failure would help
       user space to take appropriate action. For example, EMODPR
       can return "SGX_PAGE_NOT_MODIFIABLE" that helps the runtime
       to learn that it needs to run EACCEPT on that page before
       the EMODPR can succeed. Alternatively, if it learns that the
       return is "SGX_EPC_PAGE_CONFLICT" then it could determine
       that some other part of the runtime attempted an ENCLU 
       function on that page.
       It is not possible to provide such detailed errors to user
       space with mprotect().


4) #PF implementation

   (a) There is more to restricting permissions than just running
       ENCLS[EMODPR]. After running ENCLS[EMODPR] the kernel should
       also initiate the ETRACK flow to ensure that any thread within
       the enclave is interrupted by sending an IPI to the CPU, 
       this includes the thread that just triggered the #PF.        

   (b) Second consideration of the EMODPR and ETRACK flow is that
       this has a large "blast radius" in that any thread in the
       enclave needs to be interrupted. #PFs may arrive at any time
       so setting up a page range where a fault into any page in the
       page range will trigger enclave exits for all threads is
       a significant yet random impact. I believe it would be better
       to update all pages in the range at the same time and in this
       way contain the impact of this significant EMODPR/ETRACK/IPIs
       flow.

   (c) How will the page fault handler know when EMODPR/ETRACK should
       be run? Consider that the page fault handler can be called
       significantly later than the mprotect() call and that
       user space can call EMODPE any time to extend permissions.
       This implies that EMODPR/ETRACK/IPIs should be run during
       *every* page fault, irrespective of mprotect().

   (d) If a page is in pending or modified state then EMODPR will
       always fail. This is something that needs to be fixed by
       user space runtime but the page fault will not be able
       to communicate this.     

Considering the above, could you please provide clear guidance on
how you envision permission restriction to be supported by mprotect()?

Reinette