Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
References: <3e20a542e756bbc0f66435c9344ff674f3ff7ac7.1659680600.git-series.apopple@nvidia.com>
 <3f6d04570907ee80f030ffaa85e8f506b3d31fc1.1659680600.git-series.apopple@nvidia.com>
 <65a9b473-9696-04ae-862b-8d4947c22703@nvidia.com>
User-agent: mu4e 1.6.9; emacs 27.1
From:   Alistair Popple <apopple@nvidia.com>
To:     John Hubbard <jhubbard@nvidia.com>
Cc:     linux-mm@kvack.org, akpm@linux-foundation.org,
        linux-kernel@vger.kernel.org,
        "Sierra Guiza, Alejandro (Alex)" <alex.sierra@amd.com>,
        Chaitanya Kulkarni <kch@nvidia.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Felix Kuehling <Felix.Kuehling@amd.com>,
        Jason Gunthorpe <jgg@nvidia.com>,
        Logan Gunthorpe <logang@deltatee.com>,
        Miaohe Lin <linmiaohe@huawei.com>,
        Muchun Song <songmuchun@bytedance.com>,
        Ralph Campbell <rcampbell@nvidia.com>,
        David Hildenbrand <david@redhat.com>
Subject: Re: [PATCH v2 2/2] mm/gup.c: Refactor
 check_and_migrate_movable_pages()
Date:   Mon, 08 Aug 2022 11:38:13 +1000
In-reply-to: <65a9b473-9696-04ae-862b-8d4947c22703@nvidia.com>
Message-ID: <874jynzeh9.fsf@nvdebian.thelocal>
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?mVwHnwPUEaIhYapr3i4bzGnke2rkSyh7s5ctglUQJkddAdW2yXyoW+w0W18e?=
 =?us-ascii?Q?FbcBPUdTYfhgmZYK1vX9XaFnpOIE9TRW5BDFoFa88w/EXbr4LVkCGwAmCQmJ?=
 =?us-ascii?Q?FmyWzKhhFGMWugq5hp2f+kbIGhGBtALjr/HBpgjgBtMKTtyHGrYAQn2SdcVP?=
 =?us-ascii?Q?w30cMrPCVFPU12ImBRvc9dq4JS+TpCHW5LAZm+JsGfEi7l7h4AQZB7+lrUAQ?=
 =?us-ascii?Q?Oj1u2xELswOMQrgCnkURLfYxabkZK4jbyPoZCVdLQ6OkXxNTsQYPLK7ymOj1?=
 =?us-ascii?Q?5yvcQ0Tw1Ov3imdks0OF2vTKObzpUak9sd1Tr0j5ej422HmDKeLNSPxyVvHO?=
 =?us-ascii?Q?t3DckWUa9Ls9t1xMochh5Wu9Dy7w6wOlTa5uoReynuxY51ujE17B9rcOMbt4?=
 =?us-ascii?Q?W+RlcA6XLOpbXGe8405f1i1gyhT/4mw7gGaQNrOH1gRCVox38KhX118BqE1f?=
 =?us-ascii?Q?kpyb1XbrmZ76iBkhzt5d+o4MnQU5Oe19KSKWVPXWgF9isZ0oSnkutzY0EPKC?=
 =?us-ascii?Q?NGMpv4y/purx1bsFIV0qbakOBhXE92bKIBOKiuTUb0g5oudBI/YqbbvOdqbx?=
 =?us-ascii?Q?w51GAvEylHno18JcnFPztZqXU4U0t3sIQshnJ79/39HsKpUVuMUbEn1DzoCk?=
 =?us-ascii?Q?i1XlCMZweW6ViMNSqlPN5UGvzfatXYRG5twe43rr6A0J7kq3XfV0QNGvEG+C?=
 =?us-ascii?Q?7isXuOtOIUzj7HqLVB97BxkjnrQlZLqMBbXwQsedR1TZPKh5PevWD1SnV/fa?=
 =?us-ascii?Q?l5meEe2l1HtS8E93axppLJuoITpyFhZMSBx5T1c6X3jkln8nmH9CoMJplHqJ?=
 =?us-ascii?Q?Ec0LCF49TJJaoiysGeK6jcs+soK/o47S0ICmkMiYqHunaBV/IuAhlcCdCOw6?=
 =?us-ascii?Q?kbKXVh3xgVuXE8q6J01r0VKgAAvw/phCUji8V4nx2GZCp6a+WkDxAlUJ/KXQ?=
 =?us-ascii?Q?Y1WjfvHBnSFBzyro3gcy+exHTA+Y1knApjHuk8gbuZdApMPQxl5W6zmbukX4?=
 =?us-ascii?Q?4UbDG7aS3/mCIX7gD2tiidMQPj1XmGPZl/HaWfbOwSZAx8CkkQ+GEOkXEUNe?=
 =?us-ascii?Q?pKe+eYSjvZFxUPpNzhKb414FcvQARgFudNqbU9XNutDQrsoGkRyTGnf/e2kk?=
 =?us-ascii?Q?xr25Cee+VOOh0EeR5C52v8w5rbrx/Sw5CtTMeYM9yJt1uxALqfpLMGJ84RtY?=
 =?us-ascii?Q?WQD4TH7hsZUS4ry5hqf5z4A0HwkQxfvqTxQnfkG2mCX/LhvKqdidMQnUAfam?=
 =?us-ascii?Q?PnhwdmE6KswtBh7tK1vncmtfqeuzw3Usmo3gvdtorTdAWH2LxMfXhNe+5YOI?=
 =?us-ascii?Q?E6Xnljshv4dq6vw3kpjBolZL8ZUqjKwiw29du0a8q2G2zCI9XAfeP23oCFlA?=
 =?us-ascii?Q?mAdL8PpMabzDGvFuD01MPK95YWYR4evTfmdxvt/g1yKs9/1tSc0Ldv/MPvHY?=
 =?us-ascii?Q?YGPoDPI05tPA6QZerIjATpIr42Vujm/kavy547lBmlbscGDCkkfSKYQ8fGjS?=
 =?us-ascii?Q?qLnH1RLw7pYoVBV+Geg3zyHPQu1erP77XFsEjUmCwfeKGsjddmYzowkTsRN0?=
 =?us-ascii?Q?Lrze0CJ18pmFc/DKxHdgPFMuAhyS30EWpFdy27v/?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2e1f5064-d14c-4037-ceb7-08da78e10397
X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Aug 2022 01:55:05.1320
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: FOe/rIThutq6XY+DBpcNcS6O2o4Fy9UyrXQr9Yfg/y1R2fXuvsbvA0AeB3Uq4qDhO2hQOglLdwyo4LlemeKGLg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1PR12MB2445
X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


John Hubbard <jhubbard@nvidia.com> writes:

Thanks John, have fixed most of this but a couple of comments below.

> On 8/4/22 23:29, Alistair Popple wrote:

[...]

>>   +		/*
>> +		 * We can't migrate pages with unexpected references, so drop
>> +		 * the reference obtained by get_user_pages().
>
> The get_user_pages() reference is confusing, since we only handle FOLL_PIN
> here. It's hard to connect the comment to the code. Maybe a more precise
> pointer to where the reference was taken would help.

Ok. Is changing the reference to __get_user_pages_locked() better? (I
get that this is the FOLL_PIN case, but that is the function that
actually does that in __gup_longterm_locked()).

>> +		 * folio_isolate_lru() takes a reference so the page won't be
>> +		 * freed.
>
> Also confusing because it's difficult to connect the comment back to the code.
> Maybe mention where folio_isolate_lru() is called from in this case?

Will update this to:

		/*
		 * We can't migrate pages with unexpected references, so drop
		 * the reference obtained by __get_user_pages_locked().
		 * Migrating pages have been added to movable_page_list after
		 * calling folio_isolate_lru() which takes a reference so the
		 * page won't be freed if it's migrating.
		 */

I'd rather not explicitly call out the function where
folio_isolate_lru() is called as that could change. However it should be
pretty easy to find where pages are added to movable_page_list and pages
should never be on that list without being isolated anyway so this
better documents what the code assumes IMHO.

>> +		 */
>>   		unpin_user_page(pages[i]);
>> +		pages[i] = NULL;
>
> Is this correct? The loop covers all of nr_pages, so we are setting every
> pages[i] = NULL, for non-DEV_COHERENT cases. This seems wrong.

Yes, this is correct. Regardless of error or success we have to unpin
the pages, and if we've unpinned the pages we need to clear them from
the pages[] array. Will update the function description to include this.

This is also partly why I wasn't convinced the 'goto err' error handling
was better - we need to take the same actions regardless of error or not
and that perhaps makes it less clear.

[...]

>>   @@ -2049,10 +2066,10 @@ static long __gup_longterm_locked(struct mm_struct
>> *mm,
>>     		WARN_ON_ONCE(!(gup_flags & FOLL_PIN));
>>   		rc = check_and_migrate_movable_pages(rc, pages);
>> -	} while (!rc);
>> +	} while (rc == -EAGAIN);
>>   	memalloc_pin_restore(flags);
>>   -	return rc;
>> +	return rc ? rc : nr_pages;
>
> This seems to add some holes in the handling of error cases. If -EBUSY
> or -ENOMEM happens, then we return an -errno, rather than the number
> of pinned pages. But some pages may have been pinned.

No they won't - as mentioned above migrate_unpinnable_pages() will (or
at least should) always clear and unpin all pages.

 - Alistair

> Previously it just looped if there was any error, so that couldn't
> happen.
>
>
> thanks,