Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <f3cf5158-c39f-a707-a896-bef39e7ca9c3@amd.com>
Date:   Tue, 1 Feb 2022 10:09:13 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.5.0
Subject: Re: [PATCH 3/3] tools: add hmm gup test for long term pinned device
 pages
Content-Language: en-US
To:     Alistair Popple <apopple@nvidia.com>, akpm@linux-foundation.org,
        linux-mm@kvack.org
Cc:     rcampbell@nvidia.com, linux-ext4@vger.kernel.org,
        linux-xfs@vger.kernel.org, amd-gfx@lists.freedesktop.org,
        dri-devel@lists.freedesktop.org, hch@lst.de, jgg@nvidia.com,
        jglisse@redhat.com, willy@infradead.org, alex.sierra@amd.com,
        jhubbard@nvidia.com
References: <cover.516a938ce97eb805791da6e2df508eb0dce413b8.1643698773.git-series.apopple@nvidia.com>
 <d400bc4c874ff211b8520c51930b5ab78711454a.1643698773.git-series.apopple@nvidia.com>
From:   Felix Kuehling <felix.kuehling@amd.com>
In-Reply-To: <d400bc4c874ff211b8520c51930b5ab78711454a.1643698773.git-series.apopple@nvidia.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?T3J6U1YxTWhkclZJUUd3bHVRSm51Wkx2WVZ5Uks2ZUVsU2xxaEgyVWUxUFhw?=
 =?utf-8?B?Nm90Y29WU2czV09iV0d5b216bGpqWjI1SGFaU3hNZG9HS2plc25rV00zaGh1?=
 =?utf-8?B?bGlIa2g0V0wzdkplQndSSG1oQ3FUSENXd0Q2amlMNko0VThOaWd5c2luNEJP?=
 =?utf-8?B?U0diSlNzQ0dsSGkyMDRxdWo2VFdsaTA5cnBPV1EydWVOalF0M2VTeUpLNFRn?=
 =?utf-8?B?NzVIZmowUFFxWUhUSWF2Y1FnMnlIaFE1VGQvVGVMUFlLSS9rYjBhRGVhZzdh?=
 =?utf-8?B?R1NTZERRMTliMWlBVk8rQ0ttUHE4MjUyRDc3TE9RUUxaTmZXTG93UyttUk16?=
 =?utf-8?B?SmhUTzRvay8vMkpoME9DN25OVWYvWGRWYnRBV0o0d3Y4eHdKZ291ZXlEUUZu?=
 =?utf-8?B?YkhrUFZlOE9xSEgzeEtOQlhXK1JHdDZkZkFFaktiOFhNMkNmdEpNdlNwbFc3?=
 =?utf-8?B?ekI0UEVSUXF3QzZtNUlpcEx2WG5kTGxBb0N6YmcxWVdUemZJRHJpV0c4blN0?=
 =?utf-8?B?Q3cwNlBpWXBaYW9peFI3WTdmdFo2MzhoeVp4UXZFdjNsMmFxa2hudlJvUFd2?=
 =?utf-8?B?YnE2bkFxeUN4MUlZbnNhd1MwNlVLa3M4OGR2dElPN1VNOUtyQUxwSmFZbzRD?=
 =?utf-8?B?Y0RTdlRYNC9jcld6WUxwMXNxTU5sOGNWM1RRNHVZN2FrN3pZYkNaWEJKS2dU?=
 =?utf-8?B?eitNMDF5bWNZcVRQaTlPWE1ZTU51NERMMUcvL0FzSFNGUFIyYVdWalJtL2lX?=
 =?utf-8?B?a3hIT3hNdURLdHJJcUdPMDhtMDlXSkJqRnN1QlluSFJZTnd0MVBTSWdTVFVn?=
 =?utf-8?B?WlRLQ3lsd0pIRDl6TnkwTGY2YzQrZjZGbjgxL2V6WnFzU0FOVHBoMGxVTFpH?=
 =?utf-8?B?QytXdXZ4VmZSd0hHM3dUU2ZtMW1FR0ZqWU1WcGloNUxTMjNOMk92MjBHNHJu?=
 =?utf-8?B?VlJBWjFyL0dSRG94Z1ZRNEZTZS9zZUoyOFphb2FqUmJhbC9XQ0pIclJtclNY?=
 =?utf-8?B?Q1MySlg5SUxMYVg0dXo0ZVdObGhVang3R2VJK1hGdEloQzM3aG9pazBkV203?=
 =?utf-8?B?UllscGl5bDJ2N0tjVXI2VWxVVUhPVGV3YXZ0bC9wVjBRQmxwU1Eya2V3WkVI?=
 =?utf-8?B?dW5UdHJTdGY5dlpWN2toaTM2eUplZnJLNWpvMjBtTXVxYjVtTHp0MGY0MXYy?=
 =?utf-8?B?bnVBV2hoTkhIN2JWK0gvS3c1cnlWY0wwWXAyQ3BTSXNCMWVlNEk2akYxcnRv?=
 =?utf-8?B?c1VhTDJxcEhLLzZwMks0clNoNXFnWkJoN09JVlNtS0EranR3U3FwbVhQTmhj?=
 =?utf-8?B?UFZXbEI1UDJGbG9HMk9nRW9wWVc1Mmpwd25rRGxjNDhmSnBxWER4d2syNTBw?=
 =?utf-8?B?WkpUNzJGZTVKMUwxeG9yeGowVW1vaE1KeTVJdHVKbUtCNTFVaitFVm5GSXZE?=
 =?utf-8?B?R0R1TzNYRzF6RTlzUHRVOXBwWnc1YTZLZDdmcGpvVENiTnA1L0lLcGR5ajht?=
 =?utf-8?B?dXdRS2MvRUZlOWxuOXNvUFFnVWd3aWxpVXNHKzhkZ3V4aVo2bkhyZlFMQ25t?=
 =?utf-8?B?ZkpmSFcvR2dseVQvSkExZm1OUGRMZnd6b0tyZ3JrL0VrUkpYUTlsZ3p6dTFT?=
 =?utf-8?B?ZG54SWdaZlZWNS8zQ2I0cDh2QUprSHlxQXA2ZmQvazhBQVdYVHh2RDA3Q1FQ?=
 =?utf-8?B?V0VhUTRWWGJsYWxPWWJHY3EyWVpNbzBSTEZjNU1PU3c3OUgySWltYXFMNTYx?=
 =?utf-8?B?dDFTY0c2SjYxNGV4T0VDbTNGc1hWQ1ZEV3A4OFBrTU1yQytDSDFOR1pNT1Jv?=
 =?utf-8?B?eUhQMWNYSWcwdnFuMnBHSXFSaVdvQVR6NmErSG1SYnlmb3loL3BwTWFBWUc5?=
 =?utf-8?B?d04yeEZzVHFyOXVoZVRuWHZFOHAyMWwvT2tENHljSGtqWml4YU1jZS9hemVv?=
 =?utf-8?B?aHZPUTZ2eUV0M2F5eDJSQ2xwdisvT3dYRVdrckVmd2t4d3ArR2FNMm9jZkh0?=
 =?utf-8?B?YlNqb1NleU9tSUMzRDJySXNOeWc3b0VnKzAyOGJCbk1PNFI2MDlkcUdOWHlm?=
 =?utf-8?B?cFRUK3BCYTUwS2ZNVTdUMVpVMWhQanYrZk8rWkphLzNYNVovTkNyMXA4RDlP?=
 =?utf-8?B?QTV2bkcwUm12MmdkcVhTT21KM1FoNWEyK1E1SnVYWHZrcytnWlJZaDRETEUx?=
 =?utf-8?Q?FFIzQWQMOTdrpgj2udRsi0s=3D?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ec1bf838-97d4-4c33-db3c-08d9e594d02d
X-MS-Exchange-CrossTenant-AuthSource: BN9PR12MB5115.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Feb 2022 15:09:16.4588
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: mRe6l3jOUcVEpfQTSWRCAbVnrvgQ5Ej6hDOB4PZ0i6pyUVueZWIGwRE6TUO3wxDLkDvPEHQdoMXhM7Pq4WvWyQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1930
Precedence: bulk
List-ID: <linux-ext4.vger.kernel.org>
X-Mailing-List: linux-ext4@vger.kernel.org


Am 2022-02-01 um 02:05 schrieb Alistair Popple:
> From: Alex Sierra <alex.sierra@amd.com>
>
> The intention is to test device coherent type pages that have been
> called through get user pages with PIN_LONGTERM flag set. These pages
> should get migrated back to normal system memory.
>
> Signed-off-by: Alex Sierra <alex.sierra@amd.com>
> Signed-off-by: Alistair Popple <apopple@nvidia.com>

This patch is

Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>


> ---
>   tools/testing/selftests/vm/Makefile    |  2 +-
>   tools/testing/selftests/vm/hmm-tests.c | 81 +++++++++++++++++++++++++++-
>   2 files changed, 82 insertions(+), 1 deletion(-)
>
> diff --git a/tools/testing/selftests/vm/Makefile b/tools/testing/selftests/vm/Makefile
> index 1607322..58c8427 100644
> --- a/tools/testing/selftests/vm/Makefile
> +++ b/tools/testing/selftests/vm/Makefile
> @@ -142,7 +142,7 @@ $(OUTPUT)/mlock-random-test $(OUTPUT)/memfd_secret: LDLIBS += -lcap
>   
>   $(OUTPUT)/gup_test: ../../../../mm/gup_test.h
>   
> -$(OUTPUT)/hmm-tests: local_config.h
> +$(OUTPUT)/hmm-tests: local_config.h ../../../../mm/gup_test.h
>   
>   # HMM_EXTRA_LIBS may get set in local_config.mk, or it may be left empty.
>   $(OUTPUT)/hmm-tests: LDLIBS += $(HMM_EXTRA_LIBS)
> diff --git a/tools/testing/selftests/vm/hmm-tests.c b/tools/testing/selftests/vm/hmm-tests.c
> index 84ec8c4..11b83a8 100644
> --- a/tools/testing/selftests/vm/hmm-tests.c
> +++ b/tools/testing/selftests/vm/hmm-tests.c
> @@ -36,6 +36,7 @@
>    * in the usual include/uapi/... directory.
>    */
>   #include "../../../../lib/test_hmm_uapi.h"
> +#include "../../../../mm/gup_test.h"
>   
>   struct hmm_buffer {
>   	void		*ptr;
> @@ -60,6 +61,8 @@ enum {
>   #define NTIMES		10
>   
>   #define ALIGN(x, a) (((x) + (a - 1)) & (~((a) - 1)))
> +/* Just the flags we need, copied from mm.h: */
> +#define FOLL_WRITE	0x01	/* check pte is writable */
>   
>   FIXTURE(hmm)
>   {
> @@ -1766,4 +1769,82 @@ TEST_F(hmm, exclusive_cow)
>   	hmm_buffer_free(buffer);
>   }
>   
> +/*
> + * Test get user device pages through gup_test. Setting PIN_LONGTERM flag.
> + * This should trigger a migration back to system memory for both, private
> + * and coherent type pages.
> + * This test makes use of gup_test module. Make sure GUP_TEST_CONFIG is added
> + * to your configuration before you run it.
> + */
> +TEST_F(hmm, hmm_gup_test)
> +{
> +	struct hmm_buffer *buffer;
> +	struct gup_test gup;
> +	int gup_fd;
> +	unsigned long npages;
> +	unsigned long size;
> +	unsigned long i;
> +	int *ptr;
> +	int ret;
> +	unsigned char *m;
> +
> +	gup_fd = open("/sys/kernel/debug/gup_test", O_RDWR);
> +	if (gup_fd == -1)
> +		SKIP(return, "Skipping test, could not find gup_test driver");
> +
> +	npages = 4;
> +	ASSERT_NE(npages, 0);
> +	size = npages << self->page_shift;
> +
> +	buffer = malloc(sizeof(*buffer));
> +	ASSERT_NE(buffer, NULL);
> +
> +	buffer->fd = -1;
> +	buffer->size = size;
> +	buffer->mirror = malloc(size);
> +	ASSERT_NE(buffer->mirror, NULL);
> +
> +	buffer->ptr = mmap(NULL, size,
> +			   PROT_READ | PROT_WRITE,
> +			   MAP_PRIVATE | MAP_ANONYMOUS,
> +			   buffer->fd, 0);
> +	ASSERT_NE(buffer->ptr, MAP_FAILED);
> +
> +	/* Initialize buffer in system memory. */
> +	for (i = 0, ptr = buffer->ptr; i < size / sizeof(*ptr); ++i)
> +		ptr[i] = i;
> +
> +	/* Migrate memory to device. */
> +	ret = hmm_migrate_sys_to_dev(self->fd, buffer, npages);
> +	ASSERT_EQ(ret, 0);
> +	ASSERT_EQ(buffer->cpages, npages);
> +	/* Check what the device read. */
> +	for (i = 0, ptr = buffer->mirror; i < size / sizeof(*ptr); ++i)
> +		ASSERT_EQ(ptr[i], i);
> +
> +	gup.nr_pages_per_call = npages;
> +	gup.addr = (unsigned long)buffer->ptr;
> +	gup.gup_flags = FOLL_WRITE;
> +	gup.size = size;
> +	/*
> +	 * Calling gup_test ioctl. It will try to PIN_LONGTERM these device pages
> +	 * causing a migration back to system memory for both, private and coherent
> +	 * type pages.
> +	 */
> +	if (ioctl(gup_fd, PIN_LONGTERM_BENCHMARK, &gup)) {
> +		perror("ioctl on PIN_LONGTERM_BENCHMARK\n");
> +		goto out_test;
> +	}
> +
> +	/* Take snapshot to make sure pages have been migrated to sys memory */
> +	ret = hmm_dmirror_cmd(self->fd, HMM_DMIRROR_SNAPSHOT, buffer, npages);
> +	ASSERT_EQ(ret, 0);
> +	ASSERT_EQ(buffer->cpages, npages);
> +	m = buffer->mirror;
> +	for (i = 0; i < npages; i++)
> +		ASSERT_EQ(m[i], HMM_DMIRROR_PROT_WRITE);
> +out_test:
> +	close(gup_fd);
> +	hmm_buffer_free(buffer);
> +}
>   TEST_HARNESS_MAIN