Message-ID: <782d631e-6610-c971-5800-66fbba41b425@nvidia.com>
Date:   Sat, 11 Dec 2021 23:21:33 -0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.4.0
Subject: Re: [RFC] mm: introduce page pinner
Content-Language: en-US
To:     =?UTF-8?B?6IOh546u5paH?= <huww98@outlook.com>,
        Minchan Kim <minchan@kernel.org>
Cc:     =?UTF-8?B?6IOh546u5paH?= <sehuww@mail.scut.edu.cn>,
        Andrew Morton <akpm@linux-foundation.org>,
        linux-mm <linux-mm@kvack.org>,
        LKML <linux-kernel@vger.kernel.org>,
        Michal Hocko <mhocko@suse.com>,
        David Hildenbrand <david@redhat.com>,
        Suren Baghdasaryan <surenb@google.com>,
        John Dias <joaodias@google.com>
References: <20211208115250.GA17274@DESKTOP-N4CECTO.huww98.cn>
 <YbD8naaJrZQANahP@google.com>
 <TYCP286MB2066003B2045AD6ABE4C0295C0719@TYCP286MB2066.JPNP286.PROD.OUTLOOK.COM>
From:   John Hubbard <jhubbard@nvidia.com>
In-Reply-To: <TYCP286MB2066003B2045AD6ABE4C0295C0719@TYCP286MB2066.JPNP286.PROD.OUTLOOK.COM>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?c0hDa1NyNC96SjdnS1R4Q3RjWERiUHo2c3hZaThuMGZDKzNtK0lQemRiMHBN?=
 =?utf-8?B?aFpxUFRxWHpsNUlTaFEvL2ZmQmNPZkdTSXRXMHY2RE83SGhDNjNiZHFsL3ZK?=
 =?utf-8?B?amJKbExzQUJkb29leHpNMklXUi9rNXFMY3VWWU9VRkVTMFFLZ0lldDRtSlVh?=
 =?utf-8?B?WkdmUEtzejFQU2p1T3h5NDd2VjUxNVFaaml5SGFoaG9YbnNiV2FCMnZuQk5W?=
 =?utf-8?B?YjVvcmMzWUtITHlwYng2U2ZmWHdLN0FRdFFxZXF2WkJUbFZnNWdacFlYVlUw?=
 =?utf-8?B?U21yaVFIRzh6d3lWeFpVZFFuYnU3dzFqRmMvOG4xaVE3VThWV08rN25DelFm?=
 =?utf-8?B?OWVlVzE1SEJWZXhUMXhGWFRCUWsvR0VBRlFmVXRpWlVqT1pEd05JZTJYbk9s?=
 =?utf-8?B?cVZ2NnJFQ0x6Vjk5a1BwbDNmUERnN2FyUGlYYXdSbUxxODVpa1E1ZndYTlF5?=
 =?utf-8?B?TnFqU1pZT3YrSFJZblpqZndkK2xIVnFRMGRQWFY1b2cyNENGeE5OdWVjd0xJ?=
 =?utf-8?B?YkpWc3hxMjk2ZkQxUlQzUk40L2c3UTZCdXR1VjdESWplcnN4Szl3ZTUxdWYz?=
 =?utf-8?B?OFRiYWsrN2ViMUlCMGJMMzBtSTJxRmRpUjJCWjhtaWtFcW9zbUUyRVlxY3o4?=
 =?utf-8?B?bTdzLzhYK1l5NWJRK2NaWHFKbTVaMk9UU2dteVlzOWNvL28yVE94dUlUOFQv?=
 =?utf-8?B?WU1RVEtLWlA3bU1uZkMzbThRT2NRcW1Vd0Z4d2FvSkk1RzVTZ1FDZXBVTHF3?=
 =?utf-8?B?UkY2RjFlME5hdERwOEEzd1I1K3dDaDJ1bHdZT3JKbzFZUEx2M29IbjVaYkoz?=
 =?utf-8?B?ckZpY0JxSHJWYmNUeU1BTE40em5nbWRwVGs2cnpLTWxiaWdsYkdiTXJROU5k?=
 =?utf-8?B?UXhMQXAwVkhjdFZ3c0dZNFBYMCtJS3VOOGhaVDE4OU8xRVhLRlNDQWN1RTl0?=
 =?utf-8?B?NHVhRVJuKzl2R1hOYThuREJONkF2SVBpalpjNUl0RnQwaXpIeXozdEo4c3F0?=
 =?utf-8?B?UUc0ZFUvOVRRSlNFUjV4WmovS3ZMQjFMYzMvMk1HMWcvc2I4Rkw4Yyt5VFZN?=
 =?utf-8?B?L0Vjd2NiTmVsdnJnNjFxTHVYWnhkYUIwS3E1RWhhamxEREZIanBVZXZrTlZ0?=
 =?utf-8?B?Zk5UTkVBQis5Vm43eVJZbkJCaFdYci9JZlhtWTBFcG5YclkvVHUzVHdnMnhs?=
 =?utf-8?B?QURuY3ltQXM1aXhzclpDOEdhWUhLYnRuZTROck9jQVdkdnZsTi9uSWQ3eWJS?=
 =?utf-8?B?ZDYwU0RGMWUrcHVZQm5RNjduZ0hSOWpocmJRYnE4QXg4QTFsQy9ZdGlSUG9n?=
 =?utf-8?B?bGJMRTVTVUlVQU8wQmdYZzFFTmg1RjBPdHMzQmNrMEZCM0Q0aldlNnNTOFZF?=
 =?utf-8?B?SHBManVYQmUxWVhDR0dWbGN2WXVtZmdTN1RUekpjdnNUR1lJMFEzOEpVTDNK?=
 =?utf-8?B?QWd5MlFwQ3RqN1VYd2JXYmp5TWp3V3ozOHFFRGptdnBjbFdCZ2ZaS0MzMFJL?=
 =?utf-8?B?VVJmNlVtWU1PRU4yU2lEbFlmVEl3dklVLy9EM25qUW5wb3pxNHRNbk4yeDUr?=
 =?utf-8?B?b2loNGpydzZ5K2ZkY3FsZDlHemtIR3VrUGg5TUVQZ2tuaEtGS01hVFEzc1hz?=
 =?utf-8?B?MXQwSW9aYUFJZkdrTjZIakpVWCtwN3lVYURHUm12NzdJdXByZ3c0dWpCU25Y?=
 =?utf-8?B?dFF6UEd4Zml4eEh1Z1lzVGpESmlKQnUxWmZJaStWWnp0NzZ0dUFLUU5zU054?=
 =?utf-8?B?MW54MmpvWHMwZk9LQXRWVnpYbUhQVU13enI1NW1yaDdheDA3aW0veHFYVUlu?=
 =?utf-8?B?S1I0VzRCdmZkbFRqblpsc0F5YUsvQThmKzFRVkdrZUd0cmlWeGFKeks0dkVC?=
 =?utf-8?B?Z2FiMHpOdTFMVWY4clhNSFNFaWxPRW0va3puQ2FTQllSeXdDTUsvT0NmWlJ2?=
 =?utf-8?B?R3UyVHo5TlhyVmlsNmtpUDljcXZqZ21QUlpQNk1iS2VPZDAwRTFKMWk3VVp4?=
 =?utf-8?B?anJtZEpNV1hFUVRBQzBzdkJjWjFGenY5VzFIOHlKNE5QMGphS2pyeUlLeG5S?=
 =?utf-8?B?UDV3RnFoMHI4SmUrbmx0clFOZ0I2SFNlMlFKanB1WXlSekdKamE5c2lNYWU5?=
 =?utf-8?B?RFFpTmJzNHJZT0RSVCtQRXF4dHJ1TWVzQ0ZzR3pZdGE0Z1pCaldBVjRPaC96?=
 =?utf-8?Q?Di1wrM+ICf47KJ2VXf2vrhE=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: af2ee1db-87b4-449f-2996-08d9bd40075a
X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB4130.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Dec 2021 07:21:35.1510
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: /9tXL1+1CFQDc+3ZSmYE5SOsVPEgU5pUIgCJb60eTuJ04vsHX1rLOwBcdnfILPmQ3FzEQAfOoSz3v7trk4qsoA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4999
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 12/10/21 01:54, 胡玮文 wrote:
...
>> So you are suspecting some kernel driver hold a addtional refcount
>> using get_user_pages or page get API?
> 
> Yes. By using the trace events in this patch, I have confirmed it is nvidia
> kernel module that holds the refcount. I got the stacktrace like this (from
> "perf script"):
> 
> cuda-EvtHandlr 31023 [000]  3244.976411:                   page_pinner:page_pinner_put: pfn=0x13e473 flags=0x8001e count=0 mapcount=0 mapping=(nil) mt=1
>          ffffffff82511be4 __page_pinner_put+0x54 (/lib/modules/5.15.6+/build/vmlinux)
>          ffffffff82511be4 __page_pinner_put+0x54 (/lib/modules/5.15.6+/build/vmlinux)
>          ffffffffc0b71e1f os_unlock_user_pages+0xbf ([nvidia])

The corresponding call to os_unlock_user_pages() is os_lock_user_pages(). And
os_lock_user_pages() does call get_user_pages().

This is part of normal operation for many CUDA (and OpenCL) programs: "host memory"
(host == CPU, device == GPU) is pinned, and GPU pages tables set up to point to it.

If your program calls cudaHostRegister() [1], then that will in turn work its way
down to os_lock_user_pages(), and if the program is still running on the GPU, then
it's reasonable for those pages to still be pinned. This is a very common pattern
for some programs, especially for those who have tuned their access patterns and
know that most accesses are from the CPU side, with possible rare access from the
GPU.

>          ffffffffc14a4546 _nv032165rm+0x96 ([nvidia])
> 
> Still not much information. NVIDIA does not want me to debug its module. Maybe
> the only thing I can do is reporting this to NVIDIA.
> 

...or hope that someone here, maybe even from NVIDIA, can help! :)

Let me know if there are further questions, and if they are outside of the linux-mm
area, we can take it up in an off-list email thread if necessary.


[1] cudaHostRegister():
https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__MEMORY.html#group__CUDART__MEMORY_1ge8d5c17670f16ac4fc8fcb4181cb490c


thanks,
-- 
John Hubbard
NVIDIA