Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Content-Type: text/plain;
        charset=utf-8
Subject: Re: [PATCH V5 15/31] x86/sgx: Support restricting of enclave page
 permissions
From:   zhubojun <bojun.zhu@outlook.com>
In-Reply-To: <8f7c676e-952b-3409-312a-be4cadaf7194@intel.com>
Date:   Wed, 7 Sep 2022 10:39:28 +0800
Cc:     bp@alien8.de, cathy.zhang@intel.com, cedric.xing@intel.com,
        dave.hansen@linux.intel.com, haitao.huang@intel.com, hpa@zytor.com,
        jarkko@kernel.org, kai.huang@intel.com,
        linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org,
        linux-sgx@vger.kernel.org, luto@kernel.org,
        mark.shanahan@intel.com, mingo@redhat.com, seanjc@google.com,
        shuah@kernel.org, tglx@linutronix.de, vijay.dhanraj@intel.com,
        x86@kernel.org
Content-Transfer-Encoding: quoted-printable
Message-ID: <PSAPR04MB4167F203DD60585D538155C0E9419@PSAPR04MB4167.apcprd04.prod.outlook.com>
References: <PSAPR04MB416734EEED145D832A04B936E97B9@PSAPR04MB4167.apcprd04.prod.outlook.com>
 <8f7c676e-952b-3409-312a-be4cadaf7194@intel.com>
To:     Reinette Chatre <reinette.chatre@intel.com>
MIME-Version: 1.0
X-MS-Exchange-MessageSentRepresentingType: 1
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 698c356e-39a0-4e83-1dd5-08da907a90fb
X-MS-TrafficTypeDiagnostic: PSBPR04MB3910:EE_
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: Rk2XjIgBdtwhKqMybQ5tke3f4si74b1r//3QUwCDtLSSOttV989jhjTyxnCuTVWzcxpmnVfq4YrsVIJ+m2SoScPt0lfI7YwfGDt3LofFfPp2p+qNVrka5ouzvSOoFXou2W5g/DCmT4c2J+q/3m2g4FE+4ci3JWr0TCQ9v3q/BOqWhfbSdMMn/AHWYBtH9xolRGvF1OI3urRvv6AtofsESevfg5OVJqF9bSN1FCc9t/a4lm08uazYXMuLREqOeREnn2SE9iYftBDaSV1mFxm3/cn262bclCJ6R97bhk1eA089wB6CELOiBN8LZM3WTZnqX8SxPfGcAXesjiIWcHk5//ln3+3Iz7OwPttsFL1kPjaAaEAPQD5IS+qj5XTpS76u5+/XVjjHzq2YauhygHc/ysukavyYnruwdbDb8lEowqHQ5Es+QvwEAS7HtSpVhVGCh8h/W1LBekRgQg0aMb972YkUm0tllw58d9dj5ge2TLT1IJVAyxpZQLtuRJb2wjYNR+fNbd4AN0lFmp54TbKqOwF34nlrRynVL5z7ua7Hd24DyskDfmu803fJcr/3YBpBXpT7tiPQ7ul6BnngHnDoJ1ME+iBO9TvHVNTnAwI5zB/6mV+8EbJz7EY6H7rNCkT2KGRgFG0pZ/1HWuOgCJ1OKGPyN+eC39+6TnsSUZ01DPI4N0Ie41DVZn94FcjdIBIq
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SzR4S29WR21wY3FrTG9Ec3Y4R0c2TzJLOFpoVWkySHA3clZzbFh2bGF4SFdj?=
 =?utf-8?B?dXpmT0x2T3lweElZd3BjWVU0cVBwSjN4ZnVmZ3Yrai9BR09TSDM3UU1SdmVM?=
 =?utf-8?B?VDBLTm9uTDZLRE51bkhOT1B4ek5BZDBUWFIyb2ZqOE1kQlZqN2ZuUHBoUHVj?=
 =?utf-8?B?UTgwdFhXb1FZZ1dLTm1Lc1pYeWszWlFvWE9MbFFOWTVCMVJ0a09qTnlFS2V2?=
 =?utf-8?B?U3hEaURpSGRoRWxlUnd4dDM0UjVTY09XbVlIaERxTjljTUl5a0F3bmE2TlFN?=
 =?utf-8?B?TTAzc3d3OC9JWGJrekpkL3JqaFFUSGJYWUtpQXlzbGJMc2d1TlF0RENhOFZo?=
 =?utf-8?B?ZEZ5d2ROdHh1eFlHalptbVJoOENIOEFlc0l5c3R6SjQ5eit5UzZ2OVhmVzB1?=
 =?utf-8?B?ZTVOV2lWRlFUNm4vVHVYL3MvUE00enp4cC9ZUnNIOGF0OWQyTmpRN2xsWnp2?=
 =?utf-8?B?czZrbCtWdDZyVEdxbnZlbzExYmQ1bnZRb01YWEhnV1hXMlZXRnF2a05jYzBS?=
 =?utf-8?B?cGZKaVN4NWEvOXlmb2NZNUFmK0ltVjFLZTM3VmhFVnBjeDRvNDJDTFZpMnZC?=
 =?utf-8?B?YWkrUmRkcE1pbW1iR3Z6N3ZhSWNZbDNBYVVjUzl6RXNDUlpoU1hTL2VtUXJR?=
 =?utf-8?B?eVNwWHd1M3QrOVRoNmFsTEZSeU14MGk5UnZFWXdxaXJOL0FMQVlzK3ZMaTBO?=
 =?utf-8?B?aWV1cVdZK01uaXpQSGxsWW56eDZEU3BYckFOR1BpekNEZHJFSm1iQ0ltaDhX?=
 =?utf-8?B?ZkltMit0WmZxR1BrcUNTVmw4emFuay9yNHlndFVQcTFicTBpMkd2MFovbGhw?=
 =?utf-8?B?LzU4Z1FSVFM3R0FqY0xjVWE5Rms1dzY3NEJOaDJlVERVSDViNjVoY2s0RUtp?=
 =?utf-8?B?RE56d1FNUWtRZVhwRFB2dno3RlJRY3pWUjNJZlNwYUtNeTNPaWpxdmtZRWVO?=
 =?utf-8?B?amZ2RVd6Tit6eFpsaC9wdkFMT3lwWlU0R2xXYzY3b1YzNDJJaG5wWG40UXRz?=
 =?utf-8?B?TkgreGFPMWZOQU1vV3R0U21yZFlta2ZEbHptcWdTMzhsejB4a3Q4UWdLKzRE?=
 =?utf-8?B?ZmFndXJ1aFBCbW1IMnJ4azE3cWpldmdQVnU3L05ndlc3bUlwVG92S2VMRlFQ?=
 =?utf-8?B?Nk5uZzdoOUxjL2FQa0pBVzB6bnpLVEZFM3g2UXgzNWdqT1FuYnNuU2tSRjNP?=
 =?utf-8?B?U3A0OFUzbHJod1JJODVLb3huVUt0MFNMcFNaQ1NQQ3NBdWdJejIvc05EVFJH?=
 =?utf-8?B?YitWMWNuZE5WelpicEsySE1oZGkwdklrdmxwVkdad2duZkhWUlhzaEhEeW95?=
 =?utf-8?B?UTlDT0RUT3hOZThXeHZROVRDc2JkenRhRmxqdXJRY3ZxOXE3YUgzMjBLOUtp?=
 =?utf-8?B?VGU0WVY3ZGhRc0ZqOFRZN1Q3QmN1Z0FVYVBTcHBxRXhKQXBMaXVXTjdmUkdJ?=
 =?utf-8?B?d2g0RDNzWmlROWhjN0JlNGZxTXpLdStaWkhaeXpNVHZPdjZ6K2lpaWF3OE4y?=
 =?utf-8?B?STMzYXp1WjMrQ2ljTEVvelZLdmRLak5HOGJSUnQ3bjNtODJ5ZTkvVzJTUjAw?=
 =?utf-8?B?cHZZemJpY3VCVUtNY3pxWFM5ODNBYzhHQ2lveWExMzFMZDZ0djlQamF1T0JV?=
 =?utf-8?B?Nkd1N1gvb3hZdGpMOVU0QTZCNUJlcGZPNUR6eGp6d2tIY0ttekx2cGxhUVVX?=
 =?utf-8?B?SWhxVktTVDZ4WUxxdW02ZDl0YTBrZmpiNmJPOWtubExXeTA2NEZhSno2a0Mx?=
 =?utf-8?B?SEh3YjVINHRZTTdLVFFvZkhXT0V1ZFRrc1NucmJUNkJRY05PZnpTYm11SS8r?=
 =?utf-8?B?eXlVd1RCUlVEUEZJczVuQT09?=
X-OriginatorOrg: outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 698c356e-39a0-4e83-1dd5-08da907a90fb
X-MS-Exchange-CrossTenant-AuthSource: PSAPR04MB4167.apcprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Sep 2022 02:42:12.1007
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PSBPR04MB3910
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi, Reinette. Sorry for late reply! Appreciate for you detailed explanation=
!

> On Sep 2, 2022, at 23:22, Reinette Chatre <reinette.chatre@intel.com> wro=
te:
>=20
> How important is the performance of page permission restriction? How
> about the performance of page type modification?

Enclave applications may need to change its page permissions at runtime.
If they change page permissions frequently, it may introduce substantial
overhead, compared to applications running on native Linux.
(`mprotect()` is more lightweight compared to restricting and extending
page permissions inside enclave)=20

But I have not profiled the detailed of the page permission restriction=E2=
=80=99s
performance. So I=E2=80=99m not sure how much benefit we will gain for perf=
ormance
if we move the ETRACK flow outside the `for loop`.

> From the hardware perspective, a single ETRACK can be run after
> EMODPR is run on a range of pages.
>=20
> Some things to keep in mind when considering making this change:
>=20
> Note that the enclave's mutex is obtained and released every time
> an enclave page is modified. This is needed to (a) avoid softlockups
> when modifying a large range of pages and (b) give the reclaimer
> opportunity make space to load pages that will be modified.
>=20
> Moving the ETRACK flow out of the for loop implies that the mutex would
> be released between the time the page is modified and ETRACK flow is run
> (with enclave mutex held). It is thus possible for other changes
> to be made or attempted on a page between the time it is modified
> and the ETRACK flow. The possible interactions between different
> page modifications (both initiated from user space and the OS via
> the reclaimer) need to be studied if it is considered to split
> this flow in two parts.
>=20
> With the ETRACK flow done while the enclave page being modified is
> loaded there is a guarantee that the SECS page is loaded also. When
> the ETRACK flow is isolated there needs to be changes to ensure
> that the SECS page is loaded.

Thanks for pointing out such case which I have not considered yet!

> It needs to be considered how errors will be communicated to user
> space and how possible inconsistent state could affect user space. In
> support of partial success the ioctl() returns a count indicating
> how many pages were successfully modified. With the configuration
> and ETRACK done per page and their failures handled, the meaning
> of this count is clear. This needs to be considered because it is
> not possible for the kernel to undo an EMODPR. So if all (or some of) the=
=20
> EMODPRs succeed but the final ETRACK fails for some reason then
> the successful EMODPR cannot be undone yet all will be considered
> failed? How should this be reported to user space? Variations may be ok
> since EMODPR can be repeated from what I can tell but I envision scenario=
s
> where some pages in a range may have their permissions restricted
> (and thus have EPCM.PR set) but which pages have this state is not
> clear to user space. I don't know what would be acceptable here.
> Looking at the EACCEPT flow in the SDM it does not seem as though
> EPCM.PR is one of the EPC page settings that are verified.
>=20
> Reinette

I agree with you. It is hard to handle when there the final ETRACK fails
but EMODPR succeeds.

Thanks for showing the case I have not considered but
needs thinking deeply!

BR,
Bojun