Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Teterevkov, Ivan" <Ivan.Teterevkov@amd.com>
To:     Jan Kara <jack@suse.cz>
CC:     Alistair Popple <apopple@nvidia.com>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "jhubbard@nvidia.com" <jhubbard@nvidia.com>,
        "rppt@linux.ibm.com" <rppt@linux.ibm.com>,
        "jglisse@redhat.com" <jglisse@redhat.com>,
        "ira.weiny@intel.com" <ira.weiny@intel.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: find_get_page() VS pin_user_pages()
Thread-Topic: find_get_page() VS pin_user_pages()
Thread-Index: AdlsqyEC2Ib370amRP6esQRt2dSqSwAI4d6AABNzQCAAA7uWgAACkwJA
Date:   Wed, 12 Apr 2023 12:13:38 +0000
Message-ID: <PH0PR12MB56068CA4549ED998188D0623F09B9@PH0PR12MB5606.namprd12.prod.outlook.com>
References: <PH0PR12MB5606D4611050BC8B1CC430FEF09A9@PH0PR12MB5606.namprd12.prod.outlook.com>
 <87mt3ehti4.fsf@nvidia.com>
 <MW5PR12MB55984F39C8CECADDFE7548F2F09B9@MW5PR12MB5598.namprd12.prod.outlook.com>
 <20230412104151.hkl5navnaoc7l7ob@quack3>
In-Reply-To: <20230412104151.hkl5navnaoc7l7ob@quack3>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?swzfvSn2WzVyPGcSazLjlzBCCBK1aMD4OgmP0iUiSEH2O5WeDskSbcNdap1v?=
 =?us-ascii?Q?8kz6hVJRx1g7Ftp8nQBPHmawIOEW7iwcMPutPwhsTCUFqKfi3KlGQv2+TxO9?=
 =?us-ascii?Q?z8ykuHFm4D+sTj2yf7r5i4x9ih69lj+QReHLQTwqxhfP7Ga/BP9asHHteiXa?=
 =?us-ascii?Q?JFod5MAtsJrSeSlrHL1/mOew3nwykucWldfxd2G2KoocB4+OnIe1tFw0y0pN?=
 =?us-ascii?Q?oE/xKQLf3A+hwmaoSM+n9saSCwOGzwuQGcSX9kyTK82EPIfTmwo36CONjTrZ?=
 =?us-ascii?Q?NAjGucF3JXZdUx4eIj3HxtD8Hxi8//ZTPdU+ec2Lu4nY243mFT01qv/e/5CD?=
 =?us-ascii?Q?skw1QDeofNX3E3HLspESikmDvFVaB2pjYYPBBwQ+VR9o3AtQs+U5j4HcUWzg?=
 =?us-ascii?Q?P6SpdyFWj4dVH2QGmJN7MZjB/UJMmOEPQtpb06DBokU6RoliIfYCrNVFpSOD?=
 =?us-ascii?Q?vLAnd8tRsh5yR/5SW3YuyU6lvBsbzpk1/RKp6638rXaC+RXe63gNG5i9drjD?=
 =?us-ascii?Q?IB6oaf/rREIoaS6ZhLINnRiwem/IA+i5EmHoZFo53O0edf4K7ckAGNuZnWFd?=
 =?us-ascii?Q?z3jKkKsj0AMGvfXVtEIpFulJHa8RGrdBbhojUotgF4hbsoOlsXzP4bB11A6o?=
 =?us-ascii?Q?pDZsxHJ1z7sESKSyDMNtMnC4epNMnfbVMhQMPJUjUkxivgckzA2SCV/ez0ZI?=
 =?us-ascii?Q?CwS/QMqrS3nbeOovB56hBOwOVSYPw4026P2mjEvoRdSAsexOcJLXuqG5eSwd?=
 =?us-ascii?Q?9ShPZCCIGPzGUFzXMR6ObZqMAT5QCyDHS60g76Sy11xDGz062NRhm8mwYDT5?=
 =?us-ascii?Q?t97XCqQUWBOWvlnkVpX2IHJX0mXfwdsTpPrbMdoPR4FBTDGu9rJR5rPuStUH?=
 =?us-ascii?Q?peFQi45a9QHlSs3z3uWH6J+BuR/xdLVG0SVzGzbUPuEnHSBl4eABjdnuR/Ci?=
 =?us-ascii?Q?P97ivDzi6YrF8Vd1YZ6f+k0S4L5nlePz7tNjMLr1WZMq+M0J1/MuL5BZwVzK?=
 =?us-ascii?Q?/5ZbzMkHUmrLhnDg94e72jn+PDTM6A8mGnoFrxMP6tHPiwPgmIc0jUkmde4/?=
 =?us-ascii?Q?jHovIpv/aRaLcHwEgxDcN7tQjUfRjML0aVGkT2JppKTJ90RQ5DE1KdBPKF7w?=
 =?us-ascii?Q?/i0l4ylJ7JoYxERW6nR0k+baYqaab2OeCFQjRTilUeWc7GDP84VX8VX4tlzI?=
 =?us-ascii?Q?hJA2J8ebRxR/s6gC7RyMNMwAUeN+Gk7SVPw3N022EXl9fYwGmnx6VEBWWYaQ?=
 =?us-ascii?Q?WR67JvcVYk/wRTA+BgYpWjEjkqz3Zpl1JhgllR27ltJrE62rvrBxfw9eUh5L?=
 =?us-ascii?Q?xz9GkzsY9CQsocnLIjYEw3clMyBL4otwdSgzGxTP+0mMaCiDZVZyTrbXv5M7?=
 =?us-ascii?Q?awXSiR2OKw8o17D+njCq4XtxoRCfGmYABQwrndvpZUbXLjzL/iZaH76zLHML?=
 =?us-ascii?Q?DLx3WB4V3h76Q0Lj55yFR3u9ihslNnVQeP4CrxGJ3nU9LYzU9Gxqao6H80OO?=
 =?us-ascii?Q?x7+SeIpPetQ//XPW9cKUJa0S2vCbw28UlnTTNK3lLN4naD3FHLBdcEH5p8UL?=
 =?us-ascii?Q?ovjlUj1CNq02MuzjOuM=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PH0PR12MB5606.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: abeb72f1-6708-438a-e2e4-08db3b4f58d0
X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Apr 2023 12:13:38.1429
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: QIWiz2BNJBxvAW+d5V+bEugHBdZ2Z2LomfCJsyN5yMLD3iXFXJjBPfBm7q6UTWX/f3vPqEVqMndW4eBeE04DWw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4221
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

From: Jan Kara <jack@suse.cz>=20
> I'm a bit confused. Above you write that:
>=20
> "The memory allocation workflow begins in the userspace, which creates a =
new
> file backed by 2MiB hugepages with memfd_create(MFD_HUGETLB, MFD_HUGE_2MB=
)
> and fallocate(). Then the userspace makes an IOCTL to the kernel module
> with the file descriptor and size so that the kernel module can get the
> struct page with find_get_page()."
>=20
> So the memory allocation actually does happen from fallocate(2) as far as=
 I
> can tell. What guys are suggesting is that instead of passing the prepare=
d
> 'fd' to ioctl(2), your application should mmap the file and pass the
> address of the mmapped area. That's how things are usually done and it al=
so
> gives userspace more freedom over how it prepares buffers for DMA. Also t=
hen
> pin_user_pages() comes as a natural API to use in the driver.
>=20

I failed to explain that the kernel module might call vfs_fallocate() to
allocate hugepages, then find_get_page() and finally dma_map_single(), all
before the userspace maps it. Sorry for the confusion.

> Now I'm not sure whether changing the ioctl(2) is still an option for you=
.
> If not, then you have to resort to some kind of workaround as you
> mentioned. But still pin_user_pages(FOLL_LONGTERM) is definitely the API
> you should be using for telling the kernel you are going to DMA into thes=
e
> pages and want to hold onto them for a long time.
>=20

Changing the application workflow and then doing ioctl() with the address i=
s
what I ideally want with either find_get_page() alone or vm_mmap() with
pin_user_pages() as a workaround, and the latter is preferred.

Thanks,
Ivan