Received-SPF: pass (google.com: domain of linux-kernel+bounces-57386-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Message-ID: <908066c7-b749-4f95-b006-ce9b5bd1a909@oracle.com>
Date: Wed, 7 Feb 2024 18:24:52 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v3 0/3] A Solution to Re-enable hugetlb vmemmap optimize
Content-Language: en-US
To: Matthew Wilcox <willy@infradead.org>, Will Deacon <will@kernel.org>
Cc: Nanyong Sun <sunnanyong@huawei.com>,
        Catalin Marinas <catalin.marinas@arm.com>, muchun.song@linux.dev,
        akpm@linux-foundation.org, anshuman.khandual@arm.com,
        wangkefeng.wang@huawei.com, linux-arm-kernel@lists.infradead.org,
        linux-kernel@vger.kernel.org, linux-mm@kvack.org
References: <20240113094436.2506396-1-sunnanyong@huawei.com>
 <ZbKjHHeEdFYY1xR5@arm.com> <d1671959-74a4-8ea5-81f0-539df8d9c0f0@huawei.com>
 <20240207111252.GA22167@willie-the-truck>
 <ZcNnrdlb3fe0kGHK@casper.infradead.org>
 <20240207121125.GA22234@willie-the-truck>
 <ZcOQ-0pzA16AEbct@casper.infradead.org>
From: Jane Chu <jane.chu@oracle.com>
In-Reply-To: <ZcOQ-0pzA16AEbct@casper.infradead.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?dEF2bUMvN1ZVYkwrZGFSWmpOK3VuRGdpa1FyRlBReWRVNEE3bnVOcGdvM2hy?=
 =?utf-8?B?TE1odXBHai82ZkRNR2NlcU9hRnlmL01ZRklqSFljSTBielh6MUUxNFcrMDlw?=
 =?utf-8?B?R1ZQY0xyVHgzQy9Fa1AxOU1rRzRJR2lHVDkvQ0Q1VElsK3lBSlRXTExrY2Vv?=
 =?utf-8?B?bFZxdExmdklIVXZEK1RnaWNZVGtxN1dYYUh4S0hQOGdUTWxZcGVYMGxCRytH?=
 =?utf-8?B?bVZodG5kZklGdzJrSW02SnNqMTlGZHI4emUyN0h0a0hCQ3lab1ZzcmpZLzBz?=
 =?utf-8?B?dnNvZ2hFZWRYQThqTFRtT3NlWCs0eGJGSmVhdXl3TXMxS0kvWW9SSm40SHBO?=
 =?utf-8?B?TUNkSEg5dFFqMEVwUmtSQzFkeHVXeUZDYUcxOVFnQVhlM2FMWkFocVErd1NE?=
 =?utf-8?B?dUFHTW9rNHVNaGZlYzQ1dEo3bUlpYUJIWm1BUW1ROVNNZ3E5Vk1Zb1NRNFlh?=
 =?utf-8?B?TWl1SkhvNWV5UlU1OExVRmNNMzlWVDVJSm9GcTRjSzZXaXhmVndkTHMwRlBx?=
 =?utf-8?B?LzUrUjBNTkZIbHplaEk4d0EwbTc2Qy83Um1YT2k1a011R1k1ZjFhWjVtY0cr?=
 =?utf-8?B?MFZoV2o2bnVEcGdqMXVURzFEWGtBUVpSU3FtcmxLNm9KbVdiYlpodlhUa1RP?=
 =?utf-8?B?VG9PODJFcFF3b2NUTEV3cUV2RkxwNEpRZWk4azk3TUJBRlNaY09KejFoREdS?=
 =?utf-8?B?TERYbWIrRFRiYXFqQm5xaUk5UnFJRExuRFBUR0RwYVVuNEVyRHJmajhUcHhW?=
 =?utf-8?B?RGZiL1lDaVd0K2JKcU9WcmdqeUhKeWltVGxTZ0hidDVpVWFPNTRobEdLL1Rm?=
 =?utf-8?B?WTlOWWtiS1ZoUkxxay9LUFVqaUIxQlNOYTVnTER0ZjY0Q0pYMHYrZzU5QUlH?=
 =?utf-8?B?RzlvSmRqblRyMDFnOGhPNGMxT2UweTMyYWQxREVMaHNBMDlXOXl3N0FuOVN1?=
 =?utf-8?B?OFFmRityTkVnRmFDb1d4c0NSV1NSTWduWXI5NTR6UElZQUVjZm5nTnFRMFVD?=
 =?utf-8?B?OFE5eGVNbHBDWnk0a2RRcEFNMEJIMU55MjlWMWhybURpMWxHT0RmdzhWb1px?=
 =?utf-8?B?YUN0N1pITjRpWkI3VGF0OUthQjJXQUNOa1h4L1hveGQ5QlhycWJNUlNlZldB?=
 =?utf-8?B?K2FYNWJ6V0VudEJoa2pWT3djM0dMcXgzazdhMmN2Y2xZODVROFRNNHcvbkx1?=
 =?utf-8?B?NnpYaXNIYjl0ZXg2MS9wQnBydnI5U1A0THBIbkN0NUJ6MWRjblpyckQ3MGxk?=
 =?utf-8?B?bTc1YWpCZW54eXlDVk9wQkhla2FSbWtxZVRFUFY4SjB2bDhmSlNrSng5bXAx?=
 =?utf-8?B?WHdwMXVIeGg4M3dkbWU2cjJKbm9LMnRwRk1QbkJSV2tGZ0VYMEV6d3JzeVEz?=
 =?utf-8?B?ajRldHZBZHpHRUN5YjFTeHJxQUxkV2l4UUxZUnpRclZpcXdmYktreSthcDJm?=
 =?utf-8?B?eW1aVUw5NEN0KzJ2aVNaM24yRW5EbnVzTk4zSEtmUUhJOFc2M0x2dEVyRFcv?=
 =?utf-8?B?TGdhNUJzMGltT0ZqUDFJQ0tLSlZNZDFIZEF4bkYxUDI5NnFRQXZXNDQvZnlF?=
 =?utf-8?B?TVRyYmxQYTBEaS94Q0c2M2Vya1AxeCtLWnBveDcyS2Jza1R0NmVPT0dHbk1y?=
 =?utf-8?B?YzNWUldwazZRWllWaUZnNEFObGFyK2cvWWNzMVVJZlcvMDVNOG1DNTE3a3Jp?=
 =?utf-8?B?aGgvU0FuQXh4QURNSnBpWTF0MWxpMnVGWUtJc0tBUGljeHEwcXhwQll2YU5N?=
 =?utf-8?B?L2xTMUtzNTJ3TzU0Z0VMTDd4aXltaHIzajFzTVp0eGV2eGJBTkFacTRSQmV2?=
 =?utf-8?B?TlIwU3FQRzUvQmxPWm9xSkFPTjNoMTJWS1pmdDJBVUhFNkRFWmRkdHRwU2cz?=
 =?utf-8?B?aGdkZXFuWDllWFFlenAvcDNDS1psZ2VXOXY5VDVSRGFrOEw4a3JvMzlTeUM3?=
 =?utf-8?B?T0tvRmFDWUtYZTk0V1p2cmNNckFXKzlMMXpwVGI0TDBNZG9qUEJPNDBCUGow?=
 =?utf-8?B?cjNCejJNbFByKy9hdm9VVmhhcXVjaklJemQramJlY0ppNzNqTW5rNVRzck1Q?=
 =?utf-8?B?d2puYmM2ZFFhMnVFbzFiQWlZTTdPd1c3N2xyanlNenltMytYU3JqTy9WSlFU?=
 =?utf-8?Q?5C2JlsQ/zMi9FBXi1ZTTagd8f?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	nb5RjC13aVSwT+FWE2cIxqlSJJP2vD+epzKmRRdHQNSzPs7nC5agyXhHk1RWy4GMa/JjG2kJ9mF8V/wboGxvl6/tTKn9IEmBmTQTPrDdxpgaXYwKcuD8iftJyZWnoXpjopdP3Hv8x/drbvWYZ0ziD/ScZpc15CKWuhZtu0Kv5Puf/Ho6UyRFim5IhANITctUnzrPpYCibZKc2IqKPmfg7+1GhuNN7NpNCwas+guX56vVEoN4fV5Flm9KMdQ6/NyhD1Bgu95tOkeU+ebAMsCDjz15VX1xEuIAUzWNrxbBoyICCK2Jemjwed2toHsFLY8mRqQV6/wp9xwu53zOBpe0qgK5GBXGv4MR/rA9N9NQUhVTr3cp0yoUTWa3Rp+Qj0MTwqPLcru4M+GQW/CYBhR0VfXESfErdKjy7/9HdiPv5f5+wdnScY+kVsxmSUZ7MnezRpJpVlCW0Tn8nY8RDMiSCoQPpYs8S4gYjaKZSMRIrG6D45IFx4VXO4nMi8gYAkKazTC89kw6rD0oej83TEaGtuDO+xRQ6kQplm8NwEpBEjSvjsSfA//oXh3s0oVgcZdC3HdYRxpWtVt7wszuicJTFfuRFdrXJGps4IJYfVzyMWQ=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 87b5a4fe-cfe5-4776-d8ce-08dc284d250e
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4429.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Feb 2024 02:24:58.0845
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: zFMR3YFGIBItAM+PfY90RynBjjMDXZzALvf+8sT53seSDHFrYFlZrK65MqFYMi1bqZiRcK/fnXn76cPHYNJjew==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR10MB5947
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-02-08_01,2024-02-07_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 spamscore=0
 mlxlogscore=999 bulkscore=0 mlxscore=0 phishscore=0 adultscore=0
 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311290000 definitions=main-2402080010
X-Proofpoint-ORIG-GUID: EroeD8ghp4jWoaBihSvyCtTDHleQanMf
X-Proofpoint-GUID: EroeD8ghp4jWoaBihSvyCtTDHleQanMf

On 2/7/2024 6:17 AM, Matthew Wilcox wrote:

> On Wed, Feb 07, 2024 at 12:11:25PM +0000, Will Deacon wrote:
>> On Wed, Feb 07, 2024 at 11:21:17AM +0000, Matthew Wilcox wrote:
>>> The pte lock cannot be taken in irq context (which I think is what
>>> you're asking?)  While it is not possible to reason about all users of
>>> struct page, we are somewhat relieved of that work by noting that this is
>>> only for hugetlbfs, so we don't need to reason about slab, page tables,
>>> netmem or zsmalloc.
>> My concern is that an interrupt handler tries to access a 'struct page'
>> which faults due to another core splitting a pmd mapping for the vmemmap.
>> In this case, I think we'll end up trying to resolve the fault from irq
>> context, which will try to take the spinlock.
> Yes, this absolutely can happen (with this patch), and this patch should
> be dropped for now.
>
> While this array of ~512 pages have been allocated to hugetlbfs, and one
> would think that there would be no way that there could still be
> references to them, another CPU can have a pointer to this struct page
> (eg attempting a speculative page cache reference or
> get_user_pages_fast()).  That means it will try to call
> atomic_add_unless(&page->_refcount, 1, 0);
>
> Actually, I wonder if this isn't a problem on x86 too?  Do we need to
> explicitly go through an RCU grace period before freeing the pages
> for use by somebody else?
>
Sorry, not sure what I'm missing, please help.

 From hugetlb allocation perspective,  one of the scenarios is run time 
hugetlb page allocation (say 2M pages), starting from the buddy 
allocator returns compound pages, then the head page is set to frozen, 
then the folio(compound pages) is put thru the HVO process, one of which 
is vmemmap_split_pmd() in case a vmemmap page is a PMD page.

Until the HVO process completes, none of the vmemmap represented pages 
are available to any threads, so what are the causes for IRQ threads to 
access their vmemmap pages?

thanks!

-jane