Received-SPF: pass (google.com: domain of linux-kernel+bounces-165213-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Date: Wed, 1 May 2024 09:10:32 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: Christoph Hellwig <hch@infradead.org>
Cc: John Hubbard <jhubbard@nvidia.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	LKML <linux-kernel@vger.kernel.org>, linux-rdma@vger.kernel.org,
	linux-mm@kvack.org, Mike Marciniszyn <mike.marciniszyn@intel.com>,
	Leon Romanovsky <leon@kernel.org>,
	Artemy Kovalyov <artemyko@nvidia.com>,
	Michael Guralnik <michaelgur@nvidia.com>,
	Alistair Popple <apopple@nvidia.com>,
	Pak Markthub <pmarkthub@nvidia.com>
Subject: Re: [RFC] RDMA/umem: pin_user_pages*() can temporarily fail due to
 migration glitches
Message-ID: <20240501121032.GA941030@nvidia.com>
References: <20240501003117.257735-1-jhubbard@nvidia.com>
 <ZjHO04Rb75TIlmkA@infradead.org>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZjHO04Rb75TIlmkA@infradead.org>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?mDcEyuKOVjIAERu9Qxo8GmTEsJYBGtc/cDf06KkCC9JpTmGG7XgM97VH6zbs?=
 =?us-ascii?Q?Mwk39pjsAL5kZXOXp0KHliu4HEyP+EoO2iAvmedD4ezpp9w5Dk4Rn62MxvBE?=
 =?us-ascii?Q?l91uICDVp0gQpHPgFnC7/+LYUzRGG7PBEQAQINqPADSruQ3DkMLSABrNsWCF?=
 =?us-ascii?Q?xwVsF8ERED593Q0u4qyTauh0ufA/juSMFLr6kqkFAjUSdXNoeu9Jhaqsm3Lr?=
 =?us-ascii?Q?75ZGVmDH5hCG6IFp3xQDwHqMRL9m+DIOUSbs17ETfI+jpP5c27lXXFnjyolQ?=
 =?us-ascii?Q?YsxLHeR2iWddh/8nlNJ3DNBPbPv+7BDuc/t70fHkJXXs0pNlF+9G4VfWnfoL?=
 =?us-ascii?Q?TpD1ILLXE26OyyG7SSoXxIm4NgsdtdqYGnUkhj68WYQ55EviivG6ZOcee79H?=
 =?us-ascii?Q?ktDxippODFe+kx2ItaWX6qIjLOQkRG/FIEnNhulwu24kH7wWMc8MgvK60Xdf?=
 =?us-ascii?Q?QjFqdMYB/dwsV4eDw4SVJz3I7vZLZoEBVl8elJg04BEcFXAO+/g89EtlPgEM?=
 =?us-ascii?Q?beFzmw9vvwXstXyVCcDN3gBiZ9iS9GlJ7CPO15d+6MAWguhhvu+9bVf+Ck5j?=
 =?us-ascii?Q?y1N1P5GesSE8VBhCVb7cAqWzwJCleMO/SxpkakI+ehdTiwVb8rAykPF4/ARz?=
 =?us-ascii?Q?vMbVeVhAX6Beg44/WsMKUJZQbxHy7aWf6OoioYeSe/hwNx64ATlbIUwAzelt?=
 =?us-ascii?Q?XnRAuWQset5nHkwAlqxg30W9K4E0WdvDKZTo7DdQAiJYfulYBbP9nLsKYRXf?=
 =?us-ascii?Q?GwY8lY664YB9xthz8022WXrslvU1LxmbT5PBBCcG6BgU+qSIyhJqXNkrOROS?=
 =?us-ascii?Q?ZY5NwY3mSIVaDAUgecf7ABRZVyzjjiHKDAGhAjW9P9IGYqrX7hv/olIRlk1B?=
 =?us-ascii?Q?bgMvaxUPeFl9Luhf5yegVzCflux+VXhjW0o3rMXTMN6/WKB8ESUQZCgstRGR?=
 =?us-ascii?Q?nIsu768KS4msd++EAxON4G1ApWB/SC0GPtqn9hcS/JCfCWIKvf3PkwmgGtQO?=
 =?us-ascii?Q?h8gnpQ90b4hQpW2v4STeCF7483yA284rB6qJ+BSGJnspUgq+NtSl4bCBbJp5?=
 =?us-ascii?Q?uqpTfmOgD+Hbgd2C9L4RJ7kimUyOd+r7R43b6/4GWbno3aGVZzO4aVgrjGxj?=
 =?us-ascii?Q?sZCaBvgjnq2D3q0kmOxFROfL/3bOx1lBDt5XOKgXkeCp4u176crnGQOSpZmW?=
 =?us-ascii?Q?LaGcUcWxkxeEGDr29nLG1hLP6YDgcYVu+kxz0KbCJkZyFBfYtSx3uFHhaYzB?=
 =?us-ascii?Q?1iMLHEh/rMTYLMsInXT96zhKpKSj7qCiC8PVh0K2qq/ZE2ce/73GPocnwerM?=
 =?us-ascii?Q?KcFB7uQlIzPwO6WWk5msr0gRcs+qr3N4ZdZ+cuDJb7bdcVVMdJiEed3cY+hY?=
 =?us-ascii?Q?H3V9bYWYI5rHW9wthAtkrdNhB5EZ0ZMNAEn2isx4oVWB/5qqojrZJfVd/tOm?=
 =?us-ascii?Q?VjeihMQxjGRCZHV7MnTqxzyYqyMLwDXlDZE6FtrsUcxIu2ssfixSMUg0rC1U?=
 =?us-ascii?Q?OEkhnmLem71wigsApUvya33kFIlUyvs84c/PZt51Zn0PPQVov2GTzdE0div1?=
 =?us-ascii?Q?TTJqaYE65A5AzQGEnEIclox+czBaXFQEKI9aMeR+?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ad664f03-e733-4e47-5f08-08dc69d7b41a
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 May 2024 12:10:34.1953
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: V+r7PVpf1xn4kawtTN5uNQ+nybR9mNnPhUukwuhUcVqjbZxhSU+Ipm0bS4SPBbon
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7283

On Tue, Apr 30, 2024 at 10:10:43PM -0700, Christoph Hellwig wrote:
> > +		pinned = -ENOMEM;
> > +		int attempts = 0;
> > +		/*
> > +		 * pin_user_pages_fast() can return -EAGAIN, due to falling back
> > +		 * to gup-slow and then failing to migrate pages out of
> > +		 * ZONE_MOVABLE due to a transient elevated page refcount.
> > +		 *
> > +		 * One retry is enough to avoid this problem, so far, but let's
> > +		 * use a slightly higher retry count just in case even larger
> > +		 * systems have a longer-lasting transient refcount problem.
> > +		 *
> > +		 */
> > +		static const int MAX_ATTEMPTS = 3;
> > +
> > +		while (pinned == -EAGAIN && attempts < MAX_ATTEMPTS) {
> > +			pinned = pin_user_pages_fast(cur_base,
> > +						     min_t(unsigned long,
> > +							npages, PAGE_SIZE /
> > +							sizeof(struct page *)),
> > +						     gup_flags, page_list);
> >  			ret = pinned;
> > -			goto umem_release;
> > +			attempts++;
> > +
> > +			if (pinned == -EAGAIN)
> > +				continue;
> >  		}
> > +		if (pinned < 0)
> > +			goto umem_release;
> 
> This doesn't make sense.  IFF a blind retry is all that is needed it
> should be done in the core functionality.  I fear it's not that easy,
> though.

+1

This migration retry weirdness is a GUP issue, it needs to be solved
in the mm not exposed to every pin_user_pages caller.

If it turns out ZONE_MOVEABLE pages can't actually be reliably moved
then it is pretty broken..

Jason