Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Message-ID: <d7a6dedb-03c5-fad1-e112-c912473c7214@intel.com>
Date:   Mon, 8 Nov 2021 11:48:18 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Firefox/91.0 Thunderbird/91.3.0
Subject: Re: [PATCH] x86/sgx: Fix free page accounting
Content-Language: en-US
To:     Jarkko Sakkinen <jarkko@kernel.org>, <dave.hansen@linux.intel.com>,
        <tglx@linutronix.de>, <bp@alien8.de>, <mingo@redhat.com>,
        <linux-sgx@vger.kernel.org>, <x86@kernel.org>
CC:     <seanjc@google.com>, <tony.luck@intel.com>, <hpa@zytor.com>,
        <linux-kernel@vger.kernel.org>, <stable@vger.kernel.org>
References: <373992d869cd356ce9e9afe43ef4934b70d604fd.1636049678.git.reinette.chatre@intel.com>
 <6e51fdacc2c1d834258f00ad8cc268b8d782eca7.camel@kernel.org>
 <2a0b84575733e4aaee13926387d997c35ac23130.camel@kernel.org>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <2a0b84575733e4aaee13926387d997c35ac23130.camel@kernel.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?c21XRFp5dEgrNU03QXpONVZhUEQyWmtteTZRNlV0RHlsTzRwTTBtNGhIdXp5?=
 =?utf-8?B?em5oZVU4djNQdGxCSXNWSGNBWnlvcnpTeHRqUUlpV1BiY3oyN2RMY2hpUkYv?=
 =?utf-8?B?ZS9yYUFFUmtMdVdIb1NIUjBEMVpZNm9NY2J1UGNnYUZEbndwaVk1MU5BK3V1?=
 =?utf-8?B?M09UUjZESWFMdUJ1ZlhPL1M2c1ZxNWY2bks1ZzltT3BrbW1VOThrcWJIVFNV?=
 =?utf-8?B?MUsralBEb1dOQmtyckRrSE9MY1JIVEJxTVl0ODBqRTJ2aDJBQ2dpdGE0WUFR?=
 =?utf-8?B?TVNSMlNZWWFCSkRNMW1kTzdtd3pBTEl1cGY2cExPYWZKZG5DbFpOZnExNHg1?=
 =?utf-8?B?NVlaaE9uM1NMcDJ5bWtYb0l6VjJBcHdqR1ZyM2Rnd1FRVGNCM01jbThUTVIw?=
 =?utf-8?B?VFVZdjdZc2k4WVhxM2l0Q2hSUFZGbG04N0J2UXdsSTJ3a2NKMjlqS05zY2Vp?=
 =?utf-8?B?R0RvaU0yNXpSYmFvcHoyMzNBK1ZKRUhjOUthaVpBUjNYOHdRRzhrZ3FmNG5k?=
 =?utf-8?B?bkFTaUxJQTNwRkNkQlpUSHNLMVN6NWZjL2JSRnc2L2M2azJ6dnR3UUp4ZWFE?=
 =?utf-8?B?RGlJa0FOZGFpYm51bHcwdWdwckFMcEFaWWd2eFRPbHgya1JjcHVQVWVyeTVW?=
 =?utf-8?B?QldQeUhvQ2Z5M1graXdRaWttdW8vd3ROVzdta1U0NG1DVnF5RXMzOHV1WXpa?=
 =?utf-8?B?c0IxWkJmU08rTjFVZ3ZOcUV2T29kTGhxMXhZd0wrNE50akdnVEZIS1VEazll?=
 =?utf-8?B?dmRlb0NGdTNhTFJDSkttRGxOVkN5ZElWbjBLY0JubEpEcEo5MGlaZy9HUmRt?=
 =?utf-8?B?OGRZdWI4Vm04Q0prUUFCYXQ5ZzBCMDNTc0FHUzdmWTdHT2J1bS9sT2hUaEJB?=
 =?utf-8?B?NHFYVkgyY2V0Ny83YTRtMXFyVjRURi9pSTlmNjlwZ2ZvT29DcGxSNEVlZnpo?=
 =?utf-8?B?M2MzZU12RUU0YnNTbUVLUStmUzJMWmJRRzBJMk0zK0xRaTN3VlhyVVc4SjA3?=
 =?utf-8?B?eER4ejliVGhTREpMSjE5SzljQnRoOU51dTcydW9IQmZpUWl2U2h5WnFNWGYy?=
 =?utf-8?B?N2tUc0krbzNYWlQxajZhcnU5SFowbk5HSDJmckpHT2ZKbVZxOGtLM0FvWHVT?=
 =?utf-8?B?UlZqOE5RcUJQNE9BWG9QV29CMGQzc083N0tLcFVHK0wrc21TYlRMTUFTVkZ2?=
 =?utf-8?B?UXVSVjVjMUlucVN0dm1RZGFTZktCZElkUGkvWVBJMW9ReDFxcWc2L0U1bG1t?=
 =?utf-8?B?VW5QRlRjeVA0M0NHVGcvckl2N0Nxd0NOa3g5NndQMVg1bmh6Q1JMT2ZQTThN?=
 =?utf-8?B?aWxCdHp3MXowTHVVbHoxZkhmOFRTZXcvenFCYXpFUFFQU0htdmF5WTFzWnFq?=
 =?utf-8?B?NlRJOWVERUZOaFdkb1R1Mkd0VkdTQ0g5d0FwNFpDQlV6WllTWlpDR2hVM29J?=
 =?utf-8?B?QUp2dkNGMlVUdUtWQ2FqU1g4ZEVUb2I1c1pGUzZEU2lvUkp5ZVBiUnVMQlpx?=
 =?utf-8?B?MHAvS1BGb2IvOHphWi9oUGVQZUUzMWVGQmhqUHYrcGFiVTRIdXVtcjFvM2Ru?=
 =?utf-8?B?K3BuenhlT3dDY3hVZml6R2h4TytRQjQ2ZW1wSHZxR01DREVGRlJESnVrQjlx?=
 =?utf-8?B?TzRyTlFwL09wMVRPL3BDUll3eU1sYlVJRG5rM09YbC9NMU10dFcxSVA0S1J6?=
 =?utf-8?B?OEwwZVZQWVRrbVMzbmNyZ2FKbXBXQlFydGhEakNKM1BWR3NOOWh2SEdXaVBh?=
 =?utf-8?B?RVZmc002WUlBQ3hpR1pWazZSRG82RjQ0RW4wZkFVQmtXanNMWllCTGhBTXY4?=
 =?utf-8?B?Z1BBc3p4SXVLU2FvbWdWV2JRWGVtdFVaeFN3ajc5djA4Rnl0UlFYS216SlJr?=
 =?utf-8?B?ZTloTFlUeXA1aEVHTXkvT0FEZG1qeTJRTlNpeEZDczlMc096SUNrdWVQcmhz?=
 =?utf-8?B?ZW94cGZ3d2tHZ1prVjZMOEQzOCs0NnFsUkRiRGRRMVFLajRNeVJMZys1Rm84?=
 =?utf-8?B?Y2lQdml6VlFvY2FhaldhdlQ3TlY5T2RWRWJTUlppdE1DTVRqYUZoeS9SZGRC?=
 =?utf-8?B?bXZOempqejRORDBTcklIcUJXZU5wYjNsekFVNkFsQkVwVE9HSjZ5c0hZcHBW?=
 =?utf-8?B?cUhIMzN0eVAvOXphZE9FczdBRzZsbEw5TFp2ZjRYTVJSL0VhbWY3bklwTTdN?=
 =?utf-8?B?WEpOTU1uRk9GV0dvNWxKRTlseFZ1VVV0QXhvV2tuRG9sVEtLRmNHNkF0N2gz?=
 =?utf-8?B?MjlWeHFVeWx5ZlZjcmFKVVRnVDNRPT0=?=
X-MS-Exchange-CrossTenant-Network-Message-Id: d3434b01-1045-4ada-34d7-08d9a2f0b870
X-MS-Exchange-CrossTenant-AuthSource: BN0PR11MB5744.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Nov 2021 19:48:22.1600
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: qKDQsfx2wjPJEN48Ol6WAIvlvQfSBh9p+PTsAV+3VMwN8DP41SAt6UdpA/WRXtWYjVPBarfhqwCuuQNA1/ze1Uk7lkAIcJr8FsRg93lKKHw=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR11MB3553
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Jarkko,

On 11/7/2021 8:47 AM, Jarkko Sakkinen wrote:
> On Sun, 2021-11-07 at 18:45 +0200, Jarkko Sakkinen wrote:
>> On Thu, 2021-11-04 at 11:28 -0700, Reinette Chatre wrote:
>>> The consequence of sgx_nr_free_pages not being protected is that
>>> its value may not accurately reflect the actual number of free
>>> pages on the system, impacting the availability of free pages in
>>> support of many flows. The problematic scenario is when the
>>> reclaimer never runs because it believes there to be sufficient
>>> free pages while any attempt to allocate a page fails because there
>>> are no free pages available. The worst scenario observed was a
>>> user space hang because of repeated page faults caused by
>>> no free pages ever made available.
>>
>> Can you go in detail with the "concrete scenario" in the commit
>> message? It does not have to describe all the possible scenarios
>> but at least one sequence of events.


I provided significant detail regarding the "concrete scenario" in a 
separate response to Greg:
https://lore.kernel.org/lkml/a636290d-db04-be16-1c86-a8dcc3719b39@intel.com/

That message details the test that was run (the test hangs before the 
fix and can complete after the fix), the traces captured at the time the 
test hung, analysis of the traces with root cause of why the system is 
hung, traces after fix applied demonstrating why user space is able to 
make progress and explaining why the test can complete.

Unfortunately the traces I provided wrapped and are not easy to read. 
The essential message from the two traces are that the first trace 
(before the fix) shows that the system is stuck (almost 100%) in the SGX 
page fault handler and not able to make any progress and user space 
hangs. The second trace (after the fix) shows that the system splits its 
time between the SGX page fault handler and the reclaimer enabling user 
space to make progress and the test can complete.

 > I.e. I don't have anything fundamentally against changing it to
 > atomic but the commit message is completely lacking the stimulus
 > of changing anything.

The problem needing to be fixed is that sgx_nr_free_pages is not updated 
safely on systems with more than one NUMA node.

Reinette