Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Content-Type: text/plain; charset=iso-8859-15; format=flowed; delsp=yes
To:     "Jarkko Sakkinen" <jarkko@kernel.org>, dave.hansen@linux.intel.com,
        linux-kernel@vger.kernel.org, linux-sgx@vger.kernel.org,
        "Thomas Gleixner" <tglx@linutronix.de>,
        "Ingo Molnar" <mingo@redhat.com>, "Borislav Petkov" <bp@alien8.de>,
        x86@kernel.org, "H. Peter Anvin" <hpa@zytor.com>,
        "Dave Hansen" <dave.hansen@intel.com>
Cc:     kai.huang@intel.com, reinette.chatre@intel.com,
        kristen@linux.intel.com, seanjc@google.com, stable@vger.kernel.org
Subject: Re: [PATCH] x86/sgx: fix a NULL pointer
References: <CU4OBQ8MQ2LK.2GRBPLQGVTZ3@seitikki>
 <20230717202938.94989-1-haitao.huang@linux.intel.com>
 <dfb1f233-aebd-50cf-8704-e83b91ee110a@intel.com>
Date:   Tue, 18 Jul 2023 13:11:36 -0500
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
From:   "Haitao Huang" <haitao.huang@linux.intel.com>
Organization: Intel
Message-ID: <op.18ah5mn3wjvjmi@hhuan26-mobl.amr.corp.intel.com>
In-Reply-To: <dfb1f233-aebd-50cf-8704-e83b91ee110a@intel.com>
User-Agent: Opera Mail/1.0 (Win32)
Precedence: bulk

On Tue, 18 Jul 2023 09:27:49 -0500, Dave Hansen <dave.hansen@intel.com>  
wrote:

> On 7/17/23 13:29, Haitao Huang wrote:
>> Under heavy load, the SGX EPC reclaimers (current ksgxd or future EPC
>> cgroup worker) may reclaim the SECS EPC page for an enclave and set
>> encl->secs.epc_page to NULL. But the SECS EPC page is used for EAUG in
>> the SGX #PF handler without checking for NULL and reloading.
>>
>> Fix this by checking if SECS is loaded before EAUG and load it if it was
>> reclaimed.
>
> It would be nice to see a _bit_ more theory of the bug in here.
>
> What is an SECS page and why is it special in a reclaim context?  Why is
> this so hard to hit?  What led you to discover this issue now?  What is
> EAUG?

Let me know if this clarify things.

The SECS page holds global states of an enclave, and all reclaimable pages  
tracked by the SGX EPC reclaimer (ksgxd) are considered 'child' pages of  
the SECS page corresponding to that enclave.  The reclaimer only reclaims  
the SECS page when all its children are reclaimed. That can happen on  
system under high EPC pressure where multiple large enclaves demanding  
much more EPC page than physically available. In a rare case, the  
reclaimer may reclaim all EPC pages of an enclave and it SECS page,  
setting encl->secs.epc_page to NULL, right before the #PF handler get the  
chance to handle a #PF for that enclave. In that case, if that #PF happens  
to require kernel to invoke the EAUG instruction to add a new EPC page for  
the enclave, then a NULL pointer results as current code does not check if  
encl->secs.epc_page is NULL before using it.

The bug is easier to reproduce with the EPC cgroup implementation when a  
low EPC limit is set for a group of enclave hosting processes. Without the  
EPC cgroup it's hard to trigger the reclaimer to reclaim all child pages  
of an SECS page. And it'd also require a machine configured with large RAM  
relative to EPC so no OOM killer triggered before this happens.

Thanks
Haitao