Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
Message-ID: <d476aa51-f855-bfb8-0738-3190439a2f72@intel.com>
Date:   Tue, 8 Mar 2022 11:16:19 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Firefox/91.0 Thunderbird/91.6.1
Subject: Re: [PATCH v6] x86/sgx: Free backing memory after faulting the
 enclave page
Content-Language: en-US
To:     Jarkko Sakkinen <jarkko@kernel.org>, <linux-sgx@vger.kernel.org>
CC:     Dave Hansen <dave.hansen@linux.intel.com>,
        <stable@vger.kernel.org>, Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        "maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT)" <x86@kernel.org>,
        "H. Peter Anvin" <hpa@zytor.com>,
        Jethro Beekman <jethro@fortanix.com>,
        Sean Christopherson <seanjc@google.com>,
        "open list:X86 ARCHITECTURE (32-BIT AND 64-BIT)" 
        <linux-kernel@vger.kernel.org>
References: <20220303223859.273187-1-jarkko@kernel.org>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <20220303223859.273187-1-jarkko@kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZUZPRnJkYXBHYktlVmtCT0l1aENPNlBmaU43Nis1VElzSTI1RjBrZDhWd05S?=
 =?utf-8?B?S3BGOVgzZk1RMm90S0k4emRna2NhODZabTAzYS9oMFR4Wmg4MFlnUW80c0M0?=
 =?utf-8?B?cEV6WWNpeHV6bVpCS2tITVM0VGhiZDJhZmRsVWVqZGE5d0graXdDVFE4QUYr?=
 =?utf-8?B?NGxidDJLN2xDM3RhcnBmTndENFJ6TG9NdHkvckkwbmM2cXYvRUtja2prNFNL?=
 =?utf-8?B?Smk5OU1IUitBOFE5ZiszOUswL3JWWjV1OG14WVJxMlMxeGFTbFl1b1BDZ0tV?=
 =?utf-8?B?MnNnVVBPSDJRWlVDbjNtNHRYRWUyZ3NGYlVwbEEvRnJGOFRLWVVxQUlpK1Bt?=
 =?utf-8?B?Y015ZGFhdlZ5ZFk4bUVZa3lGbVdRMTZ4RzRoSFkrQUFMYTRkVFlQQThSd0xE?=
 =?utf-8?B?dWpCK2pPck5WeFdqVUlzbnpYNGN1Mi9VemFnd2wwd3VhMll2NzNwNnF5V29Q?=
 =?utf-8?B?VmY2cVlCSE9lTVZTMHB3Y2xWdFQ5R0R1TXdLbU4wSytFZW04b3o2L041azZQ?=
 =?utf-8?B?UGdOMDlFeE5WcFBwVnpqMURhZzBhU2RYSW9MamdudlIvN0dMN1Uza1VhY0x2?=
 =?utf-8?B?Y2JjN3ltRDBaRjlvcmZCbnpodVEwWkhlcGxLNFA2UStPVEhDdFBCb1l6ZktG?=
 =?utf-8?B?MDRxUGpBa1JoWHpKUlNRN29BWGVFZndDV0p5SG8wNXY0cERGdDl6N3pJVTZt?=
 =?utf-8?B?RTE1bzlaNDR0WFFHZ0Q2UUZFTTFUTmpOY0VUOEJ3dHhnWDFjWTRGckRDLytH?=
 =?utf-8?B?bDFqQzlyTFJ4VXJ5TWFtL0lYc3NHZHZZcUx4TTlNMjA4VXpzalZkYmlvek15?=
 =?utf-8?B?ZlZwbTdRN1RsdUN3YkFqcGN6SjFNQ3FWdnh4ZVZSNUlaR0FsWnRiUU12cTJG?=
 =?utf-8?B?NXFaQ1l1L01uaThrZGhYYyswYkRIbDQ5aFloWDJVNUkwUCs2SFdNRFM0RUVC?=
 =?utf-8?B?WXlEN0g1RTE3dEZQdFJQVjh1SytmVnU5YnkyRkRNenk0cDVvZ1FNYWk5OFJr?=
 =?utf-8?B?Ry9oR05OOCtZK1BlV2h4SzNsbmw0MmRFSTI1RnlCTVRxaXBTM0N2SWRrR00x?=
 =?utf-8?B?UWp0N3ZobGV0RVZVOFBpUHVwZnAySE1pekZxWTNFeUIzS1l1NXBEbHR1Ykh6?=
 =?utf-8?B?bDg0ZGkxNVA3Rm5DS2NNZnRTOWl4dHMwcjBMVmg2dTVPTjVObTU0ZDdMSTdm?=
 =?utf-8?B?bEluODFDclk4NWx1aWlra1luSVNYZkFJVldGd1JMRjJmUlpVamFkOEFaV0N0?=
 =?utf-8?B?RjRiWlhSMVlEeE1welpvcEFYZmJ0YUY5UVVHMk5wdmNwRVdaR0p3NjBZekJu?=
 =?utf-8?B?WkJheFJuY1JzNHVNVEVBUEl1b0xCbnd1ZHMwUHo1NjE5djFKZVJySUl6YWEr?=
 =?utf-8?B?YmczMlBTNVMwVkFpVUVvcW8rTHlrRW43NmVNY0QwbUwwaDZSdlJwVWxTQk1H?=
 =?utf-8?B?SWVtbHNXbnc4RExvN3ZxcGtHNzZ0T09ENmhVTnlhdEFBcWI0SkdudW56WnJo?=
 =?utf-8?B?KzZ0NTV3Ung3Y1JoenVpQmk1MWJ6N1ZQd3p0a1JZWjZySVlVbnNMRHIvM0dZ?=
 =?utf-8?B?dk1zd09hdXpRVnlJZnFJZHp0Z1dOK1NXWkJ6TUJ5cEFVSG5UZ2ZNV0orWVc1?=
 =?utf-8?B?MExMWlJ6MlgxMXpVcERmNTZuR09HTGZDOE9hWCs3VVk1YTg1U2MxdFlUV0x1?=
 =?utf-8?B?ZGlRUjBuckhoRVJvNUhTZFFIR2pTMUhqWW8vd2N5MmIxQjdsKzY5SG8ybjZo?=
 =?utf-8?B?OUNyd0h6Q0dGdlk1THhRVk5mcmlGWHFlcWo4c3BjVTljbjNhZHNobXIvYU5s?=
 =?utf-8?B?SGRJL3Fka1VPODZ2ZlZoYytDVWlPeWRrTGc3akxTektKSkNPbVQyeUVBMXJk?=
 =?utf-8?B?b01ZRGdVVzBBL1FWREt5T2o0c255NGZ2MzdCdUdvckpGVjV3VDY4MVVxTUVp?=
 =?utf-8?B?Rlc3Ky9tc2E0NlBxbE9CM2kyK0txUkh2YThmQkpxaWlVbVZVL1YvV0ZsajlV?=
 =?utf-8?B?UFl2bUxTRDd5T3pPYmF3V2xJS1o5UWtZN0J5YWdiV0xmOUw1aWlQcEtlS0FP?=
 =?utf-8?B?dzMzL2FsZ3RieGdhZTlCRFQzb0t2N2VEa1AwMVZFMXQ5a3lhd3c0bmhqQ1lE?=
 =?utf-8?B?L3lsTVp5N2FyRDZBUS8xdVZ2R3RiL2RCNmtxVkkwNDZrZDR0V1Q0eFNFSGFt?=
 =?utf-8?B?VjJ4ejBxMVVCbWJGR2Jzc3ZJdUN3QkwvU1pzMXNvUnlLdTUyd0M1czNyNFZG?=
 =?utf-8?Q?yQP6fJD3SHERUFEt25XeTeooITkVT9HXsBVizzgqKo=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 51e0acb7-b308-45cc-6948-08da01382255
X-MS-Exchange-CrossTenant-AuthSource: BN0PR11MB5744.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Mar 2022 19:16:23.5602
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: F2EjFm/kzEUHwwDVa4KZL4QElsfus8Zt6R+LMTczolEt7HNNv7egrn54pwIdUFK9Hi4kLPxdreU0Ti836wsKUdmDR3bS7PlB9PmNExkS+nk=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR11MB5250
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,RDNS_NONE,SPF_HELO_NONE,
	T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi,

On 3/3/2022 2:38 PM, Jarkko Sakkinen wrote:
> There is a limited amount of SGX memory (EPC) on each system.  When that
> memory is used up, SGX has its own swapping mechanism which is similar
> in concept but totally separate from the core mm/* code.  Instead of
> swapping to disk, SGX swaps from EPC to normal RAM.  That normal RAM
> comes from a shared memory pseudo-file and can itself be swapped by the
> core mm code.  There is a hierarchy like this:
> 
> 	EPC <-> shmem <-> disk
> 
> After data is swapped back in from shmem to EPC, the shmem backing
> storage needs to be freed.  Currently, the backing shmem is not freed.
> This effectively wastes the shmem while the enclave is running.  The
> memory is recovered when the enclave is destroyed and the backing
> storage freed.
> 
> Sort this out by freeing memory with shmem_truncate_range(), as soon as
> a page is faulted back to the EPC.  In addition, free the memory for
> PCMD pages as soon as all PCMD's in a page have been marked as unused
> by zeroing its contents.
> 
> Reported-by: Dave Hansen <dave.hansen@linux.intel.com>
> Cc: stable@vger.kernel.org
> Fixes: 1728ab54b4be ("x86/sgx: Add a page reclaimer")
> Signed-off-by: Jarkko Sakkinen <jarkko@kernel.org>

I can reliably reproduce the issue this patch aims to solve by creating
a virtual machine that has a significant portion of its memory consumed
by EPC:

qemu-system-x86_64 -smp 4 -m 4G\
 -enable-kvm \
 -cpu host,+sgx-provisionkey \
 -object memory-backend-ram,size=2G,host-nodes=0,policy=bind,id=node0 \
 -object memory-backend-epc,id=mem0,size=1536M,prealloc=on,host-nodes=0,policy=bind \
 -numa node,nodeid=0,cpus=0-1,memdev=node0 \
 -object memory-backend-ram,size=2G,host-nodes=1,policy=bind,id=node1 \
 -object memory-backend-epc,id=mem1,size=1536M,prealloc=on,host-nodes=1,policy=bind \
 -numa node,nodeid=1,cpus=2-3,memdev=node1 \
 -M sgx-epc.0.memdev=mem0,sgx-epc.0.node=0,sgx-epc.1.memdev=mem1,sgx-epc.1.node=1 \
 ...

Before this patch, running the very stressful SGX2 over subscription test case
(unclobbered_vdso_oversubscribed_remove) in this environment always triggers 
the oom-killer but no amount of tasks killed can save the system
with it always ending deadlocked on memory:

[   58.642719] Tasks state (memory values in pages):
[   58.644324] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[   58.647237] [    195]     0   195     3153      197    45056        0         -1000 systemd-udevd
[   58.650238] [    281]     0   281  1836367        0 10817536        0             0 test_sgx
[   58.653088] Out of memory and no killable processes...
[   58.654832] Kernel panic - not syncing: System is deadlocked on memory

After applying this patch I was able to run SGX2 selftest
unclobbered_vdso_oversubscribed_remove ten times successfully.

Tested-by: Reinette Chatre <reinette.chatre@intel.com>

Reinette