Received-SPF: pass (google.com: domain of linux-kernel+bounces-186615-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Wed, 22 May 2024 15:30:46 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: Alex Williamson <alex.williamson@redhat.com>
Cc: Andrew Jones <ajones@ventanamicro.com>, Yan Zhao <yan.y.zhao@intel.com>,
	kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
	kevin.tian@intel.com, yishaih@nvidia.com,
	shameerali.kolothum.thodi@huawei.com, Peter Xu <peterx@redhat.com>
Subject: Re: [PATCH] vfio/pci: take mmap write lock for io_remap_pfn_range
Message-ID: <20240522183046.GG20229@nvidia.com>
References: <20230508125842.28193-1-yan.y.zhao@intel.com>
 <20240522-b1ef260c9d6944362c14c246@orel>
 <20240522115006.7746f8c8.alex.williamson@redhat.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240522115006.7746f8c8.alex.williamson@redhat.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?Ts/je87qkGFkKqkpI/dgucj3P3dS7g14SVrK2kdpnXOFEvHKcjQj0sswP0L3?=
 =?us-ascii?Q?t1QzJPWxH+XycVULMChbF7r2alwF74dxWAvt1zR5ARJKumYO1BrpqhZMre8O?=
 =?us-ascii?Q?i2itjtPua23FmSDfntQHdMR2n0UE/63aL2a+I54mXac5PG6H85G+mkgbFr95?=
 =?us-ascii?Q?Dnu0UJfrkSwyM8ZB9Rjk6VcWGAFC8kuLbhuO0vddGDrkKWpxfDOiQ7zms1+E?=
 =?us-ascii?Q?8QUV2H6yK69Z2WL6reH09MjwYDuRWE+D2vl85hhGOi76TIU1Ekr8HIW618L6?=
 =?us-ascii?Q?Re/itWcNpNBgN0Kap17knm+JeUlr10o/iMGXO4V8OD5knDBC9G7PjNr6MtUM?=
 =?us-ascii?Q?Qwa9SJFJf35rLZC6Mu+bV2d8o5A+QlaShdH5R/Ve91HZ6x5qi6U0DSC+EjAx?=
 =?us-ascii?Q?M6mEtbGvXq5KyFlKbuK2kw6K1Cqepn77VzyfM5Y+SM0IPOTrNAdKUjbjfndd?=
 =?us-ascii?Q?QzEJa9C4LcyMD+kUEvnw1SmuG+8Z1Vp6X3UhIvniVyes2JSIc4fXUmYtuTZU?=
 =?us-ascii?Q?QjiZZs2u3h3kVfrozrtS1AJeVIVo9VYOijAqu+sAzyRAhKy1NWU+Ujo1wDP+?=
 =?us-ascii?Q?h8WONO4k7Y3CmUYdRmhA0sYecJNz4thAqgeGRTPInqUemjOINHZQH3F/saBP?=
 =?us-ascii?Q?xw29taDPPUPYp/UJUjY8oGUs3ZA+g8aex9vbTvR0EyKlziRpLwNNOeKs9m/D?=
 =?us-ascii?Q?5uXSHQm2z/wQOYeI1W7UbHzGsdsUoo75JUzO8VAx/itPgRd+1JCoqDgzmaN9?=
 =?us-ascii?Q?DmxbMh8+gT+S6auYxzAIPrDkqZnDkMLjiIg6PjisF9lhTGkxd9dbawhvaAQN?=
 =?us-ascii?Q?6CsDZsiwJ84GPKtOazlSBhFIPwU55fV8A4LBONDYWl8521eAD7ZIj9i2JbBP?=
 =?us-ascii?Q?4d4AQGkPLqRiptGUPrQWFxj7kEYsQNod5y0JaULfQN6KDrQHnX9mCMw/tLN+?=
 =?us-ascii?Q?1NGNhxcXTdIJkKLygJz5E7VTBYLVgG8R7HVbxREYLrJoCo1TPbsBHnuFMjnZ?=
 =?us-ascii?Q?rPdwBwvIvyeubxGardcmTc2SsaoHGPqhMLzC9qWIEwAy2TcNwRCTB8azqtdC?=
 =?us-ascii?Q?mLN4DajKn6s2gsosYsBAxYa8NxMBADOH/kA+0cb4exOXLOBWLhsI3n0GJZcb?=
 =?us-ascii?Q?ttAlRagqRNTRwWSYPNHhOGBUOJXO2Aueb5ssNGbIrYkeuTArKHoz5/4kpawF?=
 =?us-ascii?Q?6aXg91Ofn5I1m4qI1nngiPxe0T0mlFl1IqFXMQKUzGCfhRaHK9NI5yBuG9cX?=
 =?us-ascii?Q?23UYJQAFpMa/MC9XWHyGDBYKri8Pf0McZDVDCHQDo2DXh5MdVnZWcfhQ/M1E?=
 =?us-ascii?Q?+lWBc4AArU1pzQq8QLz+BL97t8KIr9T7lLnr1ky3WT+a51uVR4NtqoxHTO4j?=
 =?us-ascii?Q?k5QFArOUgDW2GZueU8Y0e3hJdNtuB03jmFcxgjoq5ISPZlwMuSbeFuvYbZKO?=
 =?us-ascii?Q?jLdwYcWV9bxEZkG2CRa3TOi/JX0gWup5cxqUqK1UkAyT81xMfzYuHbQAhueQ?=
 =?us-ascii?Q?ASI6FWLEUTdtJR2pZ7YiTIrRVRsYBs/HlFIVESvLd+u9MYKKXhGbtOy/WXC2?=
 =?us-ascii?Q?7VqTMa5HTviESOO8cWo=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 7aff4a2a-63e2-4bfb-ae09-08dc7a8d4d45
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 May 2024 18:30:48.6313
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: e5muDwOhkRljfq0SVgd+3oqdgSKIkR5v/9hvmXLjAiGsXATF3o6U6fDMlEpvINZr
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB6757

On Wed, May 22, 2024 at 11:50:06AM -0600, Alex Williamson wrote:
> I'm not sure if there are any outstanding blockers on Peter's side, but
> this seems like a good route from the vfio side.  If we're seeing this
> now without lockdep, we might need to bite the bullet and take the hit
> with vmf_insert_pfn() while the pmd/pud path learn about pfnmaps.

There is another alternative...

Ideally we wouldn't use the fault handler.

Instead when the MMIO becomes available again we'd iterate over all
the VMA's and do remap_pfn_range(). When the MMIO becomes unavailable
we do unmap_mapping_range() and remove it. This whole thing is
synchronous and the fault handler should simply trigger SIGBUS if
userspace races things.

unmap_mapping_range() is easy, but the remap side doesn't have a
helper today..

Something grotesque like this perhaps?

	while (1) {
		struct mm_struct *cur_mm = NULL;

		i_mmap_lock_read(mapping);
		vma_interval_tree_foreach(vma, mapping->i_mmap, 0, ULONG_MAX) {
			if (vma_populated(vma))
				continue;

			cur_mm = vm->mm_struct;
			mmgrab(cur_mm);
		}
		i_mmap_unlock_read(mapping);

		if (!cur_mm)
			return;

		mmap_write_lock(cur_mm);
		i_mmap_lock_read(mapping);
		vma_interval_tree_foreach(vma, mapping->i_mmap, 0, ULONG_MAX) {
			if (vma->mm_struct == mm)
				vfio_remap_vma(vma);
		}
		i_mmap_unlock_read(mapping);
		mmap_write_unlock(cur_mm);
		mmdrop(cur_mm);
	}

I'm pretty sure we need to hold the mmap_write lock to do the
remap_pfn..

vma_populated() would have to do a RCU read of the page table to check
if the page 0 is present.

Also there is a race in mmap if you call remap_pfn_range() from the
mmap fops and also use unmap_mapping_range(). mmap_region() does
call_mmap() before it does vma_link_file() which gives a window where
the VMA is populated but invisible to unmap_mapping_range(). We'd need
to add another fop to call after vma_link_file() to populate the mmap
or rely exclusively on the fault handler to populate.

Jason