Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Date:   Thu, 17 Feb 2022 20:19:35 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Felix Kuehling <felix.kuehling@amd.com>
Cc:     David Hildenbrand <david@redhat.com>,
        Alistair Popple <apopple@nvidia.com>,
        Christoph Hellwig <hch@lst.de>,
        Alex Sierra <alex.sierra@amd.com>, akpm@linux-foundation.org,
        linux-mm@kvack.org, rcampbell@nvidia.com,
        linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org,
        amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org,
        jglisse@redhat.com, willy@infradead.org
Subject: Re: [PATCH v6 01/10] mm: add zone device coherent type memory support
Message-ID: <20220218001935.GN4160@nvidia.com>
References: <beb38138-2266-1ff8-cc82-8fe914bed862@redhat.com>
 <877d9vd10u.fsf@nvdebian.thelocal>
 <20220216020357.GD4160@nvidia.com>
 <6156515.kVgMqSaHHm@nvdebian>
 <98d8bbc5-ffc2-8966-fdc1-a844874e7ae8@redhat.com>
 <20220216122600.GG4160@nvidia.com>
 <bf16195e-2570-3687-2b53-3f597ebfcfec@amd.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <bf16195e-2570-3687-2b53-3f597ebfcfec@amd.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?JfsDVCLqXXf6G/NbQyGXqKUwhuXemavJTXOuAHlCq1Dk0zVr2mF+g623LDsI?=
 =?us-ascii?Q?GCIrCNuvFs3Y8HGaAPhaA8icB/HQm9pyJr23gjga+yk9z4PPLKiJK2fYAoOp?=
 =?us-ascii?Q?GwJ59FThbnNbO9VLfFmE59KKVWjkJlJeqvzEBu2o06VR9kms4WVkgbxuaYpO?=
 =?us-ascii?Q?0NfWnrhtXBbwYHTZyo3F4Dy7IC8fr/wgZ827G688bSNl6bLwiIYHOJ5Apju8?=
 =?us-ascii?Q?murAD/aDk97bYuTIQKKA1A6UXmbjlOdZqVW6jamMjtpQAHqLiJtyeTVqY48p?=
 =?us-ascii?Q?7Un7K1Qmv4rCwSJxZUC5dIat2hk/qypEGAKxnS8C6wUCY1lcSqV5rzdrUSzW?=
 =?us-ascii?Q?+JR6yJCh+QlRiE/eEhkn72ADz11XJ46JGas13nGkPwMxwXLqn4oL5cCQLTIE?=
 =?us-ascii?Q?zXag7haXgHA9gRceRcLcj/h0QGI0bSJlURq2WQLuXkc/ZpRvyYmnBRgrawtk?=
 =?us-ascii?Q?oygWgtB/S3zTIJ79awZ0WyxsIH8VXgICvEJy1A8iFr8rpMS56T8t8XNTaYgV?=
 =?us-ascii?Q?PgmhB4UYVyoXRsnOQKUsBtmpKtWkp8gGEB8SqqYCR/BKhTtmmD32MOuxt3CL?=
 =?us-ascii?Q?KggIDtTaNV5OU3bF/eZRltTwjO+md6oKzpjQ//N41nObv9fnaKJIK74kQPYx?=
 =?us-ascii?Q?PtsyHn8RICsOwUylUR39Iu7elahwyB2gnh/mfEfguMhBCC5U2E6kSTsBF/vX?=
 =?us-ascii?Q?suYnIFprQuLoI6Jg7q8r9lDKmNrHA3YRxMNDjASFahT9u4XATLNHiiQ8/N/I?=
 =?us-ascii?Q?yKX/RYEU8IivQiQhz0/mUpJQY0icvBHzu4UTZH+1JJ9pxr61tfW3rZ9YFWMp?=
 =?us-ascii?Q?303VEP7aRbwityDyLafsCJTFPLg6QrgJRl2sdAR9jvKMmFZNEF8q44O4AGw2?=
 =?us-ascii?Q?HwEC/MReeqKv4ZH4WTASj80mXkic09W27L/eZ96fSOKZCJm8YJzykinTYN3R?=
 =?us-ascii?Q?3yVBVfZZtVyFMPuDFgATxc+X/M40XoUgCTf2tlzq4YQvpeI7WfE5YjlFnsm5?=
 =?us-ascii?Q?1Y1cgFxVvqK9UKNzjqwcMFXJBNTCFwTpzHFo8rUrrM7XKE1Z1d2FIXtPJ4Uv?=
 =?us-ascii?Q?rkEwtgnbpLsz8ULpi5y4QeP+4ZnJtpJ35Ret3EtcrqXvk2SEqRzHrwXS0wOZ?=
 =?us-ascii?Q?V1CBBGimLKA/Y8G9uMLdft/6KniOdpU9jNBVfBY65lAVIpPESG7iqZVPVAXW?=
 =?us-ascii?Q?G11DSvn33sr/JphtiCUSbQlrYLPzoGRC1lyQjM1FdYEaQjz++OHQma9klere?=
 =?us-ascii?Q?Gci3ifUEkhmZxChHfNVW8RYFactJfct/2yXRoOxtX//il3KzLEnwdeVzPH2A?=
 =?us-ascii?Q?O7ftw1ckeV1cY4MuLs+yceB+2FwkrBK88YyGxvhjCb2inSo2+p5XmdielpRz?=
 =?us-ascii?Q?uuGItmKXNBteybJ3M5bHP1K+F5FYfSoUDyd9o5Dns7aUDF0FGtTE8MTSGlq3?=
 =?us-ascii?Q?YA0rlGKmhJI8nxhWrZhiwM7cBgmj8G5BrUnLyTqGlyxTr3VUpB7d8OiZMtxh?=
 =?us-ascii?Q?u9W8TUFpeLPo4jSCEzZxNML+5gmVTdlJ9bL+hjmgniZ78I5t9HIC03UMf2e1?=
 =?us-ascii?Q?kJldyem8MdXwLAnVTb0=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 682bd472-d920-4442-5e15-08d9f27458c4
X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB4192.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Feb 2022 00:19:37.1863
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 15kAOkAMdyf1tOP9+DUAYY0eoMj1Fkp6/kuUcKyzqfaiue52Vvlm9rCMktn4yQJI
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4817
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-ext4.vger.kernel.org>
X-Mailing-List: linux-ext4@vger.kernel.org

On Thu, Feb 17, 2022 at 04:12:20PM -0500, Felix Kuehling wrote:

> I'm thinking of a more theoretical approach: Instead of auditing all users,
> I'd ask, what are the invariants that a vm_normal_page should have. Then
> check, whether our DEVICE_COHERENT pages satisfy them. But maybe the concept
> of a vm_normal_page isn't defined clearly enough for that.

I would say the expectation is that only 'page cache and anon' pages
are returned - ie the first union in struct page

This is because the first file in your list I looked at:

static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
				unsigned long addr, unsigned long end,
				struct mm_walk *walk)

{
		page = vm_normal_page(vma, addr, ptent);
[..]
		if (pageout) {
			if (!isolate_lru_page(page)) {

Uses the LRU field, so this is incompatible with all the other page
types.

One mitigation of this might be to formally make vm_normal_page() ==
'pte to page cache and anon page' and add a new function that is 'pte
to any struct page'

Then go through and sort callers as appropriate.

The 'pte to page cache and anon page' can detect ZONE_DEVICE by
calling is_zone_device_page() insted of pte_devmap() and then continue
to return NULL. This same trick will fix GUP_fast.

Jason