Received-SPF: pass (google.com: domain of linux-kernel+bounces-165894-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
References: <20240501003117.257735-1-jhubbard@nvidia.com>
 <ZjHO04Rb75TIlmkA@infradead.org> <20240501121032.GA941030@nvidia.com>
User-agent: mu4e 1.10.8; emacs 29.1
From: Alistair Popple <apopple@nvidia.com>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: Christoph Hellwig <hch@infradead.org>, John Hubbard
 <jhubbard@nvidia.com>, Andrew Morton <akpm@linux-foundation.org>, LKML
 <linux-kernel@vger.kernel.org>, linux-rdma@vger.kernel.org,
 linux-mm@kvack.org, Mike Marciniszyn <mike.marciniszyn@intel.com>, Leon
 Romanovsky <leon@kernel.org>, Artemy Kovalyov <artemyko@nvidia.com>,
 Michael Guralnik <michaelgur@nvidia.com>, Pak Markthub
 <pmarkthub@nvidia.com>
Subject: Re: [RFC] RDMA/umem: pin_user_pages*() can temporarily fail due to
 migration glitches
Date: Thu, 02 May 2024 11:05:46 +1000
In-reply-to: <20240501121032.GA941030@nvidia.com>
Message-ID: <87r0el3tfi.fsf@nvdebian.thelocal>
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?tL5SS4eKB9Vlduc7yw0nsBG4eLx5U6e/mdh+8Ut+9qR69uvG05K2hRweXfGF?=
 =?us-ascii?Q?7sK3GDzJHeBaQM8PU7F/gdcMShKvIgDCX1ZGHldaOeHLvUI4sTNoovYiAIcL?=
 =?us-ascii?Q?a0jvjObcfpYTuPnefupUaTlEMZwQ1voMl3RUg/mvhetzYKM7qs+pUy9S7lEE?=
 =?us-ascii?Q?tn+y9wMqSzBNTwA+lcVabn6EQRUn1sl0MRx+czreBxoVhVOIU66ZItKsR0lC?=
 =?us-ascii?Q?GhVjvRm7WH/YnNbzLK/vfcV2vxdBasBNI6aGYvIz0hMzWSJ+AvmbhjnWF8zg?=
 =?us-ascii?Q?uTziuMfVxPHRwWLbSuR3wAX/nfONLWjss6z+t+dvgm552w2MlYekKD7KW2pc?=
 =?us-ascii?Q?lEGd4SgUec8+qe79R+OTxxPGq163x9e2NKjqpN5DAT6v7R9WVjqyKvdJinWt?=
 =?us-ascii?Q?ImNUl8tBsDoB11qAjWLhYLlD8I186nUnzcuC8MQ3YXzUU+Zl1fjlOla0O+40?=
 =?us-ascii?Q?rcfdHKJhsFtV6v+iBjSRl+de8Hq/jHRK2TAsga0NGzpnuwhRqkBav39GdUMY?=
 =?us-ascii?Q?2aOxphAF8bxmZOGuRE0tpI26ilJPn8tqDOiVe4/TZARZHYF8uN8tFvNrW+Um?=
 =?us-ascii?Q?RBn/LT4G3KKgHUm/1Eh1zoScLlHSHHxZbKnMuE5x1qAx/wNyJYIFEH5RkbUn?=
 =?us-ascii?Q?P4yRNNHJRy/8Ht72Buy0tdeFdaJDetdhj/bqFCBrYRFRw5K2XJ/DxeLbJRvE?=
 =?us-ascii?Q?hegsAouhHZftMeQAaJVkCrgqmq7iaW6lyl4MnYzGMFdvGVfavZUV0pdqe7A7?=
 =?us-ascii?Q?VEkNCfFP0ZU91SrRJ2TOVRSEgCOMLJpkEzicFruzB4DoFuQQSJnFXwtmfSWI?=
 =?us-ascii?Q?0+vZD8v0iFXU5CAc+uTLRJKnNqL7CmLnFbEIl26cEV6nD9aq/ic0RgQaJoaV?=
 =?us-ascii?Q?ZLvjYtO0aBKu10Rbz9p1n+6FTsvKlApeRVnJhIgqEU7MD/JBpuNTx8L+fXrY?=
 =?us-ascii?Q?qvZK1LWGl1+22apJGhJM36GzWP9xVdxO2YHrE97iYM3SKjh2iDywaSSm3UZo?=
 =?us-ascii?Q?mIwnPIaRCyxjnRnf5qnI/v0HnqKfq8MPnP/y6mdEJIdHpzO4YBS4LtJ4w+q6?=
 =?us-ascii?Q?aQwy0uL43FKXp7uDbbfybERCEmMr+FfCPnOk9WFGC0B21Mo4ciAFFS9MSeV6?=
 =?us-ascii?Q?68LMV/WMZ9wpZ645gli63+IOBPr9sLznehK2TWv6WPKILeftImLDqNk26vaD?=
 =?us-ascii?Q?iMLpZLbvTSb1rsPa7fr2UxVHKGSBlRX6A/SmCJzIJIUTXY6uj7W9aGo6uCjQ?=
 =?us-ascii?Q?LiwjMZa28W06C49lOQXA0wWAXnoyI3vTo6S44E0KK5svYMOYqvzcaNEBWS8x?=
 =?us-ascii?Q?2zOUcUA/6975qsQZ6Xus0c6QBJzhOn9kPg9x2mPizBn6PlVoOAYeQVbnKd76?=
 =?us-ascii?Q?w3ZWSOC+ImAMD3kyN80QJ2slpD1s6xzfbU1d5BkgGmPWioUYQl+hMDBLY30O?=
 =?us-ascii?Q?M37JPIIEovs/bP4SImu/8wGWH4t5cZA041IXJ58rz+oVL22eLB/1Gkw+adgp?=
 =?us-ascii?Q?oxx/kf+FlMlyoshSn3X7MDpFMS76yda8PYlRWI5SU0bW3rAtO1FGlAEdi0nV?=
 =?us-ascii?Q?cw5UlS91AjwXwR/nLn0Qr9tzPDgC2LntV7X9lHo3?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6df38e2a-4635-4470-580d-08dc6a45682e
X-MS-Exchange-CrossTenant-AuthSource: DS0PR12MB7726.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 May 2024 01:15:51.4285
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Jr/A8ATBP+mu5jt56wJOpPsR3ABGeLXJTshxeabHMVDoyxGo93KqRhwU5Rec19m6Wtu5yTqrMVj8MKQWfynO5w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4436


Jason Gunthorpe <jgg@nvidia.com> writes:

> On Tue, Apr 30, 2024 at 10:10:43PM -0700, Christoph Hellwig wrote:
>> > +		pinned = -ENOMEM;
>> > +		int attempts = 0;
>> > +		/*
>> > +		 * pin_user_pages_fast() can return -EAGAIN, due to falling back
>> > +		 * to gup-slow and then failing to migrate pages out of
>> > +		 * ZONE_MOVABLE due to a transient elevated page refcount.
>> > +		 *
>> > +		 * One retry is enough to avoid this problem, so far, but let's
>> > +		 * use a slightly higher retry count just in case even larger
>> > +		 * systems have a longer-lasting transient refcount problem.
>> > +		 *
>> > +		 */
>> > +		static const int MAX_ATTEMPTS = 3;
>> > +
>> > +		while (pinned == -EAGAIN && attempts < MAX_ATTEMPTS) {
>> > +			pinned = pin_user_pages_fast(cur_base,
>> > +						     min_t(unsigned long,
>> > +							npages, PAGE_SIZE /
>> > +							sizeof(struct page *)),
>> > +						     gup_flags, page_list);
>> >  			ret = pinned;
>> > -			goto umem_release;
>> > +			attempts++;
>> > +
>> > +			if (pinned == -EAGAIN)
>> > +				continue;
>> >  		}
>> > +		if (pinned < 0)
>> > +			goto umem_release;
>> 
>> This doesn't make sense.  IFF a blind retry is all that is needed it
>> should be done in the core functionality.  I fear it's not that easy,
>> though.
>
> +1
>
> This migration retry weirdness is a GUP issue, it needs to be solved
> in the mm not exposed to every pin_user_pages caller.
>
> If it turns out ZONE_MOVEABLE pages can't actually be reliably moved
> then it is pretty broken..

I wonder if we should remove the arbitrary retry limit in
migrate_pages() entirely for ZONE_MOVEABLE pages and just loop until
they migrate? By definition there should only be transient references on
these pages so why do we need to limit the number of retries in the
first place?

 - Alistair

> Jason