Received: by 2002:a25:e74b:0:0:0:0:0 with SMTP id e72csp1688610ybh; Thu, 23 Jul 2020 15:31:32 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxa1JXLK2WVcb3ILgTHracYoHg4IcdUBEz0Eb04Z4OagNjFVbZBq19MzDzu5FFSHTmhymWl X-Received: by 2002:a17:907:205c:: with SMTP id pg28mr6151945ejb.22.1595543492276; Thu, 23 Jul 2020 15:31:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1595543492; cv=none; d=google.com; s=arc-20160816; b=kCgM43s3mKyHAHtMZPY0GBwcT/kAS7a8ZKWRVEEMq+JU6ZUNg18fcDn9Og3GSOeHNH vR8I5+6yPMQMMnPI86GxcLCy+N27w5FOR6zWfOtSYEHe/y+YtAqdyvNROOznwwACTxj/ aDwz9WwmtZBYGm1ZnoIqHvPoXBfXFmVdwX9y92eqnkZ1SWeq3X6e4+HN2978LOoRNsBT v33T6PXHjxY9Up3FR96OKy04i5yw6qJOqmduZnORazLV1iL6P9HU70JCIlO+MTBV9krU AenkkglcPECb1p9bF4HOmFF1mC6tdmBHfFJ/u5Zmolf/2QHy+17Sv9taj91/Oe5surN3 dDrw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:dkim-signature:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=BeVUOTJgQepW+LMWyDGWqpRY0jrU7O9ftBuhCh+HHBg=; b=k9p//AYf7Tybym37F0i4n3XNA5cfIWa9bl4FvOgPHqEP23ePmAmi3YV2WMg1f95IuG QIBzVoFsQ0kqLTA9Ax8wGBSU6vkZwkt4pcNfK7UbFgVxGXwYKM6vLPvTgkNQbSdcYdo4 ymvXzYamlrJCDJgN6DU0m0hrPIH6wQmDKVgBk8pigIViGHY9bqQ+6mD8QEtlKMUNKXcC 8LRWZOfDdc3rQ1clTdhYpc6+sk+Ij8t5z2Iph3iPIgs8ABVl+YQilzeOUUFat41Grckz dHHIl3cx2zoBhBw3jEhU9lBHnoIxWHyKLDUue8FGo/zjz+tzgMYlRrw+GRwkY+wNIEMO Uc9g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@nvidia.com header.s=n1 header.b=c5NVsizi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id bl8si2755498ejb.425.2020.07.23.15.31.09; Thu, 23 Jul 2020 15:31:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@nvidia.com header.s=n1 header.b=c5NVsizi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727938AbgGWWax (ORCPT + 99 others); Thu, 23 Jul 2020 18:30:53 -0400 Received: from hqnvemgate24.nvidia.com ([216.228.121.143]:13862 "EHLO hqnvemgate24.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727861AbgGWWav (ORCPT ); Thu, 23 Jul 2020 18:30:51 -0400 Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate24.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Thu, 23 Jul 2020 15:28:48 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Thu, 23 Jul 2020 15:30:50 -0700 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Thu, 23 Jul 2020 15:30:50 -0700 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 23 Jul 2020 22:30:42 +0000 Received: from rnnvemgw01.nvidia.com (10.128.109.123) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Thu, 23 Jul 2020 22:30:42 +0000 Received: from rcampbell-dev.nvidia.com (Not Verified[10.110.48.66]) by rnnvemgw01.nvidia.com with Trustwave SEG (v7,5,8,10121) id ; Thu, 23 Jul 2020 15:30:41 -0700 From: Ralph Campbell To: , , , , , CC: Jerome Glisse , John Hubbard , Christoph Hellwig , Jason Gunthorpe , "Andrew Morton" , Shuah Khan , "Ben Skeggs" , Bharata B Rao , "Ralph Campbell" Subject: [PATCH v4 5/6] mm/hmm/test: use the new migration invalidation Date: Thu, 23 Jul 2020 15:30:03 -0700 Message-ID: <20200723223004.9586-6-rcampbell@nvidia.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200723223004.9586-1-rcampbell@nvidia.com> References: <20200723223004.9586-1-rcampbell@nvidia.com> MIME-Version: 1.0 X-NVConfidentiality: public Content-Transfer-Encoding: quoted-printable Content-Type: text/plain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1595543328; bh=BeVUOTJgQepW+LMWyDGWqpRY0jrU7O9ftBuhCh+HHBg=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:MIME-Version:X-NVConfidentiality: Content-Transfer-Encoding:Content-Type; b=c5NVsiziP4cirzdcWC6K4LcVP4KSZN9WPHjiDB+wb369XeSgW+x0ffbg5/GsZEdHZ VNlqkYRVXriPkI7ArKZLO64O9s9bii6xIeiWVSn4lGPyTmw4PRgI+Gtqh+QLzLZuZj YWQRDbCiFOqf/ouBw74NnHOkciuxCGS3m1CrofSqd3zATJF3aqCFPcZZXNR6CqiIZz KekhEI0xT5i1MApDLFN41+rGbDo1hdbh0//Zubm+a3DuFfV5TO6Xv8RGAzhZEjAhEO j5W7XxfbfN+/isxrdbiUbqx1idn5RUo2XnM3MonM0s/2zPxlLTN2gpqZZau7Ejo0tO MGzZ+BIT5US8A== Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Use the new MMU_NOTIFY_MIGRATE event to skip MMU invalidations of device private memory and handle the invalidation in the driver as part of migrating device private memory. Signed-off-by: Ralph Campbell --- lib/test_hmm.c | 30 +++++++++++++++----------- tools/testing/selftests/vm/hmm-tests.c | 18 ++++++++++++---- 2 files changed, 31 insertions(+), 17 deletions(-) diff --git a/lib/test_hmm.c b/lib/test_hmm.c index e78a1414f58e..e7dc3de355b7 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -214,6 +214,14 @@ static bool dmirror_interval_invalidate(struct mmu_int= erval_notifier *mni, { struct dmirror *dmirror =3D container_of(mni, struct dmirror, notifier); =20 + /* + * Ignore invalidation callbacks for device private pages since + * the invalidation is handled as part of the migration process. + */ + if (range->event =3D=3D MMU_NOTIFY_MIGRATE && + range->migrate_pgmap_owner =3D=3D dmirror->mdevice) + return true; + if (mmu_notifier_range_blockable(range)) mutex_lock(&dmirror->mutex); else if (!mutex_trylock(&dmirror->mutex)) @@ -693,7 +701,7 @@ static int dmirror_migrate(struct dmirror *dmirror, args.dst =3D dst_pfns; args.start =3D addr; args.end =3D next; - args.pgmap_owner =3D NULL; + args.pgmap_owner =3D dmirror->mdevice; args.flags =3D MIGRATE_VMA_SELECT_SYSTEM; ret =3D migrate_vma_setup(&args); if (ret) @@ -983,7 +991,7 @@ static void dmirror_devmem_free(struct page *page) } =20 static vm_fault_t dmirror_devmem_fault_alloc_and_copy(struct migrate_vma *= args, - struct dmirror_device *mdevice) + struct dmirror *dmirror) { const unsigned long *src =3D args->src; unsigned long *dst =3D args->dst; @@ -1005,6 +1013,7 @@ static vm_fault_t dmirror_devmem_fault_alloc_and_copy= (struct migrate_vma *args, continue; =20 lock_page(dpage); + xa_erase(&dmirror->pt, addr >> PAGE_SHIFT); copy_highpage(dpage, spage); *dst =3D migrate_pfn(page_to_pfn(dpage)) | MIGRATE_PFN_LOCKED; if (*src & MIGRATE_PFN_WRITE) @@ -1013,15 +1022,6 @@ static vm_fault_t dmirror_devmem_fault_alloc_and_cop= y(struct migrate_vma *args, return 0; } =20 -static void dmirror_devmem_fault_finalize_and_map(struct migrate_vma *args= , - struct dmirror *dmirror) -{ - /* Invalidate the device's page table mapping. */ - mutex_lock(&dmirror->mutex); - dmirror_do_update(dmirror, args->start, args->end); - mutex_unlock(&dmirror->mutex); -} - static vm_fault_t dmirror_devmem_fault(struct vm_fault *vmf) { struct migrate_vma args; @@ -1051,11 +1051,15 @@ static vm_fault_t dmirror_devmem_fault(struct vm_fa= ult *vmf) if (migrate_vma_setup(&args)) return VM_FAULT_SIGBUS; =20 - ret =3D dmirror_devmem_fault_alloc_and_copy(&args, dmirror->mdevice); + ret =3D dmirror_devmem_fault_alloc_and_copy(&args, dmirror); if (ret) return ret; migrate_vma_pages(&args); - dmirror_devmem_fault_finalize_and_map(&args, dmirror); + /* + * No device finalize step is needed since + * dmirror_devmem_fault_alloc_and_copy() will have already + * invalidated the device page table. + */ migrate_vma_finalize(&args); return 0; } diff --git a/tools/testing/selftests/vm/hmm-tests.c b/tools/testing/selftes= ts/vm/hmm-tests.c index b533dd08da1d..91d38a29956b 100644 --- a/tools/testing/selftests/vm/hmm-tests.c +++ b/tools/testing/selftests/vm/hmm-tests.c @@ -881,8 +881,9 @@ TEST_F(hmm, migrate) } =20 /* - * Migrate anonymous memory to device private memory and fault it back to = system - * memory. + * Migrate anonymous memory to device private memory and fault some of it = back + * to system memory, then try migrating the resulting mix of system and de= vice + * private memory to the device. */ TEST_F(hmm, migrate_fault) { @@ -924,8 +925,17 @@ TEST_F(hmm, migrate_fault) for (i =3D 0, ptr =3D buffer->mirror; i < size / sizeof(*ptr); ++i) ASSERT_EQ(ptr[i], i); =20 - /* Fault pages back to system memory and check them. */ - for (i =3D 0, ptr =3D buffer->ptr; i < size / sizeof(*ptr); ++i) + /* Fault half the pages back to system memory and check them. */ + for (i =3D 0, ptr =3D buffer->ptr; i < size / (2 * sizeof(*ptr)); ++i) + ASSERT_EQ(ptr[i], i); + + /* Migrate memory to the device again. */ + ret =3D hmm_dmirror_cmd(self->fd, HMM_DMIRROR_MIGRATE, buffer, npages); + ASSERT_EQ(ret, 0); + ASSERT_EQ(buffer->cpages, npages); + + /* Check what the device read. */ + for (i =3D 0, ptr =3D buffer->mirror; i < size / sizeof(*ptr); ++i) ASSERT_EQ(ptr[i], i); =20 hmm_buffer_free(buffer); --=20 2.20.1