Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <c381db29-d74a-4986-db04-fbd35b8d80b0@intel.com>
Date:   Wed, 31 Aug 2022 11:08:47 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.2.0
Subject: Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error
Content-Language: en-US
To:     Jarkko Sakkinen <jarkko@kernel.org>
CC:     <linux-sgx@vger.kernel.org>,
        Haitao Huang <haitao.huang@linux.intel.com>,
        Vijay Dhanraj <vijay.dhanraj@intel.com>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Paul Menzel <pmenzel@molgen.mpg.de>,
        "Thomas Gleixner" <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>,
        "Borislav Petkov" <bp@alien8.de>,
        "maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT)" <x86@kernel.org>,
        "H. Peter Anvin" <hpa@zytor.com>,
        "open list:X86 ARCHITECTURE (32-BIT AND 64-BIT)" 
        <linux-kernel@vger.kernel.org>
References: <20220830031206.13449-1-jarkko@kernel.org>
 <20220830031206.13449-2-jarkko@kernel.org>
 <1f43e7b9-c101-3872-bd1b-add66933b285@intel.com>
 <Yw6/iTzSdSw/Y/VO@kernel.org>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <Yw6/iTzSdSw/Y/VO@kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dmVqRWFZbTJVZVMzTzNObW4yV0RtdDdnRUtwN1FwWnkybFNMb21GL1F3cUx6?=
 =?utf-8?B?dDdyajhPMG9Pak5HZ2pnanl2eVVkbnQ0NjFqUVRnZkVaYm9sdysvZGg5WWcx?=
 =?utf-8?B?dE1MTnRFalBOVitEZmNEZWp5TWFLYm5VY0JzVXJndHUwczFISFFObWZZb0lk?=
 =?utf-8?B?MnVMRkkySFRkcjZUUXR4L2RNakpTNjJJRDRPc3pmOVlCSFJJYUtSYTJ3U3dz?=
 =?utf-8?B?RFlBV1JNeFcxWHFZQXFVRG5JOFpLUTRkcTVPaEJCOWtXa1lqZktKVkYwYnl0?=
 =?utf-8?B?ZFJvQmRpSUU1WE9ubTZ2WXYwaUtYS1Q1aytab2tSZGt3TGovUUR1UGkzNkRj?=
 =?utf-8?B?M3NiSlNDa3FQRDBpdStOQWtXNzdBSUdmOXVmcnhTdGJ6bjVzZ2xkeFM5VkFK?=
 =?utf-8?B?NDRvYXlJTUp5U0FOdDBmN3pIc1FSOWJrTXVpL0JJNiszZG5mSWpTZjhlYUVr?=
 =?utf-8?B?YTJ0eVV3eG5QbVh1TUdtWG9kdEs0QU5TSC9KaTRZNkV2Wmk0OUNFdGZ2N21k?=
 =?utf-8?B?UXY1dUJlR1FXeEM1SWVYb1ZyMkVJWm5ncXpXU3lJeGRkVW1jOENLSXdvQ09P?=
 =?utf-8?B?OEJrY0QrWTQ0TkVLOE1hVjh6NlIycVFoMFJNUHpUSnBOUDBtVzh2YWxBUXha?=
 =?utf-8?B?UlprN2lxNEpPT0MzVExnMjFNMXoxaWtuTlJvS2lNdzlBTUxvMWtlMC9NNWZi?=
 =?utf-8?B?ZUpmakYvTDk4ZjNBTzFYZ0o3aUZ6anp6aHVvYlY4NjMxUUdTbTRVeHRGdTlo?=
 =?utf-8?B?Q3RuNk85NGllelBjeEs3UWhoNjJqcUovVWYwdHlWWlUrczJWUVBXbmdMZEd6?=
 =?utf-8?B?Mys4RE9YKzZLdU5pbnlPeENGQkxIU0M2RnF4VHpxMU5XTlZZODJaeFZhTmNu?=
 =?utf-8?B?WWxvU3hoclVxeVdCQk0yMXhpMHl3aDNXRmhWS25TTE9HWk54WTVBczhHYllt?=
 =?utf-8?B?WU5adTJ2S3V1T0V4Mjd4cXRER1ZESS9PSlB5OUxrZXVuQmdVRVY3MUVuYWdu?=
 =?utf-8?B?ZUFWQlBnUStlSEVYYVp1dGRnT0h4bS9wTjIzWFN3MGhXb1NTdGZubUt3S0Z3?=
 =?utf-8?B?R1JpdndrdHpQVEJ5QXNFMEJhc29rcjJhaVFQeUZnUGlKeTV3ZlEwQ2pKdHNj?=
 =?utf-8?B?MVhCZ1o2eEVCU2hycnE0MG1DZDVnU1VVekJ5QUZ2OUdWbytsUnZtdVkzRmlO?=
 =?utf-8?B?eG81ZCtIQXI0aGtiK2lIcWw1eGNvcFFkc1hNU1lZQlU0RXhuT3VyK2ludWtp?=
 =?utf-8?B?em5uT3dvTmRPY25MbldNSjhjdUFKOXVXNHFvWjd4bzB6S1Z4UzFMcVd0K0Rx?=
 =?utf-8?B?Vm5HZ0dYZEh6TFhMQzRLYnVVNDMwM1VnRmE2M1V5MUViZXQ1cy9HZ05SYUQw?=
 =?utf-8?B?WHU2V05Ic1dDR3d2QjlCS1M2OUNheUQzUzRNMGhwOUFiSDJBZE5KOFQyM1J4?=
 =?utf-8?B?T2s2a2ZzYXZWQXZVT1pCZUJJWVg4QXZPN2M4TUdMY0lhcmJzckxkRFZvZ1lx?=
 =?utf-8?B?NE51OUk5QWxqb2MxcDZheFJDMzhEVlQwbW4wem9DcXIwMTI0UTRuVmY0dE5T?=
 =?utf-8?B?RlNRN0hWbVlwT01GbUViODFRc1hKOGU2Y3NBR0hrbUpyanZYbzF3d3ZjMDRn?=
 =?utf-8?B?VWNicW9wenRLdHpxYnUvMEl5Z1VRb0g0bkRpUVRBd3p3RWZndzFGNThvMFFS?=
 =?utf-8?B?RG1NOGkyV1RmM0NzY01qNklxNVd6aUtJdXQvV0tsSEVudGFnblRoSW1VMUVE?=
 =?utf-8?B?ckNzSlNOekVZaEdJeEd1ZHdPa3U2WDNKelE5eHpWc3pRVGIxb2l5ZEc0SHhz?=
 =?utf-8?B?VFFXRElpcGtmV0lWYWFIVHBZS044a0FiRHVDdDFoUEQ5dHBwNDVERld0Vzcy?=
 =?utf-8?B?TG5HdWM5Y2RQYTRPcUV0REM5MGRFdm92N1d3anRzbmowbzNucHgwbXM0Sjla?=
 =?utf-8?B?QVhrTnI5V0xrUVRzVWdLUmdINjI1bitsSnhQUi8vRHcvZ25rSnA3WHFxd000?=
 =?utf-8?B?RUsxTEpjejliYjVEdW1UWXY2dEZzQlJxSjd6Wnd5RUxyWTRKUDNxb2tMK1pB?=
 =?utf-8?B?NXVwenhTZFNmQmdURlhaRFE1dFphcjNqSE1JbDlJQ2ZrZlBnZEtiWEIrT043?=
 =?utf-8?B?ZlFrbS9iOFdUMmQ4U29KRXh1S0kzT01ING5ENUg1UGFyRW5sZVJhTnN0U1Fs?=
 =?utf-8?B?eUE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 15f6dfcf-5173-412c-b0ac-08da8b7bdb5e
X-MS-Exchange-CrossTenant-AuthSource: CY4PR11MB1862.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Aug 2022 18:08:50.5719
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: e/f5SkBWkovJePWMMiVzGevdZYgHMDzrcQxnEsj2bgsBxOLqCPfn9I3YkzY6c7T6bztlHY+FmKflndiIoFAFOexV01/jxeHq5vQosWTCHdQ=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB3146
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_HI,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Jarkko,

On 8/30/2022 6:55 PM, Jarkko Sakkinen wrote:
> On Tue, Aug 30, 2022 at 03:54:27PM -0700, Reinette Chatre wrote:
>> Hi Jarkko,
>>
>> On 8/29/2022 8:12 PM, Jarkko Sakkinen wrote:
>>> In sgx_init(), if misc_register() for the provision device fails, and
>>> neither sgx_drv_init() nor sgx_vepc_init() succeeds, then ksgxd will be
>>> prematurely stopped.
>>
>> I do not think misc_register() is required to fail for the scenario to
>> be triggered (rather use "or" than "and"?). Perhaps just
>> "In sgx_init(), if a failure is encountered after ksgxd is started
>> (via sgx_page_reclaimer_init()) ...".
> 
> This would be the fixed version of the sentence:
> 
> "
> In sgx_init(), if misc_register() fails or misc_register() succeeds but
> neither sgx_drv_init() nor sgx_vepc_init() succeeds, then ksgxd will be
> prematurely stopped. This may leave some unsanitized pages, which does
> not matter, because SGX will be disabled for the whole power cycle.
> "
> 
> I want to keep the end states listed and not make it more abstract.
> 
> The second sentence addresses the remark below.

Thank you for capturing this. It sounds good.

> 
>> To help the reader understand the subject of this patch it may help
>> to explain that prematurely stopping ksgxd may leave some
>> unsanitized pages, but that is not a problem since SGX cannot
>> be used on the platform anyway. 
>>
>>> This triggers WARN_ON() because sgx_dirty_page_list ends up being
>>> non-empty, and dumps the call stack:
>>>
>>
>> Traces like below can be frowned upon. I recommend that you follow the
>> guidance in "Backtraces in commit mesages"(sic) in 
>> Documentation/process/submitting-patches.rst.
>>
>>> [    0.268592] WARNING: CPU: 6 PID: 83 at
>>> arch/x86/kernel/cpu/sgx/main.c:401 ksgxd+0x1b7/0x1d0
> 
> Is this good enough? I had not actually spotted this section before but
> nice that it exists. Apparently has been added in 5.12.

The timestamp should be removed, it is among the things listed as
"distracting information". In this case the backtrace and registers within
the trace are not adding value but I think it is important to mention
the kernel version somewhere for folks to be able to interpret the
line number provided.

Yet I see you kept the whole trace in V2 ?

>>> diff --git a/arch/x86/kernel/cpu/sgx/main.c b/arch/x86/kernel/cpu/sgx/main.c
>>> index 515e2a5f25bb..903100fcfce3 100644
>>> --- a/arch/x86/kernel/cpu/sgx/main.c
>>> +++ b/arch/x86/kernel/cpu/sgx/main.c
>>> @@ -49,17 +49,20 @@ static LIST_HEAD(sgx_dirty_page_list);
>>>   * Reset post-kexec EPC pages to the uninitialized state. The pages are removed
>>>   * from the input list, and made available for the page allocator. SECS pages
>>>   * prepending their children in the input list are left intact.
>>> + *
>>> + * Contents of the @dirty_page_list must be thread-local, i.e.
>>> + * not shared by multiple threads.
>>
>> Did you intend to mention something about the needed locking here? It looks
>> like some information is lost during the move to the function description.
> 
> Nothing about the locking that concerns the parameter, as the
> sentence defines clear constraints for the caller.

ok

> 
>>
>>>   */
>>> -static void __sgx_sanitize_pages(struct list_head *dirty_page_list)
>>> +static int __sgx_sanitize_pages(struct list_head *dirty_page_list)
>>>  {
>>>  	struct sgx_epc_page *page;
>>> +	int left_dirty = 0;
>>
>> I do not know how many pages this code should be ready for but at least
>> this could handle more by being an unsigned int considering that it is
>> always positive ... maybe even unsigned long?
> 
> I would go for 'long'. More information below.
> 
>>
>>>  	LIST_HEAD(dirty);
>>>  	int ret;
>>>  
>>> -	/* dirty_page_list is thread-local, no need for a lock: */
>>>  	while (!list_empty(dirty_page_list)) {
>>>  		if (kthread_should_stop())
>>> -			return;
>>> +			break;
>>>  
>>>  		page = list_first_entry(dirty_page_list, struct sgx_epc_page, list);
>>>  
>>> @@ -92,12 +95,14 @@ static void __sgx_sanitize_pages(struct list_head *dirty_page_list)
>>>  		} else {
>>>  			/* The page is not yet clean - move to the dirty list. */
>>>  			list_move_tail(&page->list, &dirty);
>>> +			left_dirty++;
>>>  		}
>>>  
>>>  		cond_resched();
>>>  	}
>>>  
>>>  	list_splice(&dirty, dirty_page_list);
>>> +	return left_dirty;
>>>  }
>>>  
>>>  static bool sgx_reclaimer_age(struct sgx_epc_page *epc_page)
>>> @@ -388,6 +393,8 @@ void sgx_reclaim_direct(void)
>>>  
>>>  static int ksgxd(void *p)
>>>  {
>>> +	int left_dirty;
>>> +
>>>  	set_freezable();
>>>  
>>>  	/*
>>> @@ -395,10 +402,10 @@ static int ksgxd(void *p)
>>>  	 * required for SECS pages, whose child pages blocked EREMOVE.
>>>  	 */
>>>  	__sgx_sanitize_pages(&sgx_dirty_page_list);
>>> -	__sgx_sanitize_pages(&sgx_dirty_page_list);
>>>  
>>> -	/* sanity check: */
>>> -	WARN_ON(!list_empty(&sgx_dirty_page_list));
>>> +	left_dirty = __sgx_sanitize_pages(&sgx_dirty_page_list);
>>> +	if (left_dirty)
>>> +		pr_warn("%d unsanitized pages\n", left_dirty);
>>>  
>>>  	while (!kthread_should_stop()) {
>>>  		if (try_to_freeze())
>>
>>
>> Reinette
> 
> We need to return -ECANCELED on premature stop, and number of
> pages otherwise.
> 
> In premature stop, nothing should be printed, as the number
> is by practical means a random number. Otherwise, it is an
> indicator of a bug in the driver, and therefore a non-zero
> number should be printed pr_err(), if that happens after the
> second call.
> 

Good point.

> Oh, sorry, I forgot one thing. The devices should be actually
> deinitialized in the error case. We do not want to leave a
> broken driver running.

Is this not already done on the error path of sgx_init()?

Reinette