Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 18 Feb 2022 15:26:40 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Felix Kuehling <felix.kuehling@amd.com>
Cc:     David Hildenbrand <david@redhat.com>,
        Alistair Popple <apopple@nvidia.com>,
        Christoph Hellwig <hch@lst.de>,
        Alex Sierra <alex.sierra@amd.com>, akpm@linux-foundation.org,
        linux-mm@kvack.org, rcampbell@nvidia.com,
        linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org,
        amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org,
        jglisse@redhat.com, willy@infradead.org
Subject: Re: [PATCH v6 01/10] mm: add zone device coherent type memory support
Message-ID: <20220218192640.GV4160@nvidia.com>
References: <beb38138-2266-1ff8-cc82-8fe914bed862@redhat.com>
 <877d9vd10u.fsf@nvdebian.thelocal>
 <20220216020357.GD4160@nvidia.com>
 <6156515.kVgMqSaHHm@nvdebian>
 <98d8bbc5-ffc2-8966-fdc1-a844874e7ae8@redhat.com>
 <20220216122600.GG4160@nvidia.com>
 <bf16195e-2570-3687-2b53-3f597ebfcfec@amd.com>
 <20220218001935.GN4160@nvidia.com>
 <2f2569d5-df0d-7975-7f4a-2d85ceaf29ef@amd.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <2f2569d5-df0d-7975-7f4a-2d85ceaf29ef@amd.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?tb8Ss23B/5GGqLSWKUKtwJNhQOXNDDNVa/1vtJiJImH+RpGh+1Zh2IkcT4iq?=
 =?us-ascii?Q?ZDGEPBTPP6H3/RqQV5u/Ilw2//ArBxI1GDDGzDU3D09MUV033GGTv+yxMd3p?=
 =?us-ascii?Q?r8bDEr9RwBq/fjZKf2Qz3rfa+YRuDl0P8e2+O7VFx75xWaficXNqpt1zM+8P?=
 =?us-ascii?Q?jkQ8Z6bqL5I6I9O9YpcAp0Vxxhjr1DawU9lVVn3vP0IKdiY/xaUvdO/7cIVr?=
 =?us-ascii?Q?x1+TMlOlKPvjmg1Lb0tlb7x2rTlUzJpc7y2vW5tJoX9B3NxcxyZuDM5r4ZWf?=
 =?us-ascii?Q?0lOyFbKwVMWKIi/ATGiQPPxPI5SHXZ4coCG4+vM0K55mJ4nO/Q6huCST/GVr?=
 =?us-ascii?Q?3yD6dfZGaLP5vxpRLwSqw5sdTlS9q1w7AlH3a+LiBlvgUGmmQmsDTkEtBQ6Q?=
 =?us-ascii?Q?/BJl8U5TySYgFfBCwapelJVtK+hzrZpxMlApMA6RH2bhdHDgFnfaQ5MrMNcx?=
 =?us-ascii?Q?7Ixzr9WNc+HXsyzsbXycP0F7vzp+p6Fs92iCQUrdbuYfvTu7y8ZsPXOn+mr2?=
 =?us-ascii?Q?NK/Ba4dryMo7c9Lx638D1SpWinrVsK+fwnwv52JTRxbs5YSGnYzTADK3EeQ3?=
 =?us-ascii?Q?raDTgXc97ycUnqjaQI/rYkbcVcDWvv1KdmElylr8h0G7MgmZlVeTO5ZxLDzB?=
 =?us-ascii?Q?5Bge2JdwNQGETqbpX4jSgTeQgG1oDCcejA4/UBmxCrd3CJf8+aavlvzgrQQV?=
 =?us-ascii?Q?dF8CpcUnXV9HIhkaBkR2fzC6C7vCDOobja3KY641OaNI+OZarVEgCHHdCnYt?=
 =?us-ascii?Q?742ECnrbKkwJqHBKU0ao8UBD0krkUfYMDQPyCWAOFF/OZGE186mCvb5THDFR?=
 =?us-ascii?Q?a1Gtt2qPJJUKCCv0aKQKWH17W2UUcXjrCyN/apn+tFlyVA88Vmo3nn7A244w?=
 =?us-ascii?Q?khzrj28tNPiRss8GFtAZAFcWQDbDElBa194xayTrBSsusTZz06XBxxvneNcE?=
 =?us-ascii?Q?h+qbvPTMZjV969jk5H2wmJt6osK1cdmMhzI7nzXUm0Vj/97VfKSOHqf2vPQF?=
 =?us-ascii?Q?/Ib1GJrLiu9IOFnx5WVBKmhzDHYQonEdQCRL1VeYWKbr+ZdN1MNebhQoJYxO?=
 =?us-ascii?Q?ZjC+U8T4mVlTnoAdRTBQQnRaP4vAqE7PPp4mJnVxeNcuH64s6pYQd0JP9kha?=
 =?us-ascii?Q?ovyJK+9gBA9hVb2P3tcJosiLliqfRTfDU6icFfojf20SbiSM/TgtAwhZMpqg?=
 =?us-ascii?Q?3zFKNAtuq1yRdF5m289ascyBqvGos831W+740aJxcC72g4R1lvI1yFrY5DCN?=
 =?us-ascii?Q?erIO7bLQNU9QAFPbyJf/I3HNTeHnQOA4noDjbb9XMzZ1jUbhnZpIcjMIQ8pZ?=
 =?us-ascii?Q?b8Px2Hx4ktdJXD654Zp5OAcXoTb0YC6E2bynCs12L/B3zsGqcC9aof/GvTV8?=
 =?us-ascii?Q?/TW5IT38QNO3hjZM1r5ME4PGvq19MI9Gqs7tzHMe3GAftExPOjC5tn9Pmfhb?=
 =?us-ascii?Q?jIc+Xv6WCN7nFnSBn+BBZbqUGerG8X56GJo4fIrJhHdBdukChqjo7H21hwbQ?=
 =?us-ascii?Q?6eW1OiAFaE91icjEXbJkuTL825q7Rh6LPKmGxszYTLoHdMkf9kAXn5YlkC+e?=
 =?us-ascii?Q?xlxjGIzCh6KlUj/qfzs=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e22f5591-89e3-47ce-c4c5-08d9f314974e
X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB4192.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Feb 2022 19:26:41.4459
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: otEe4ccNStw3oaFmyA1DVOCwzgwUd2Gv3li4CPtcuOKLBemYT6N08bAlV3m5MZ5y
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4608
X-Spam-Status: No, score=-1.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-ext4.vger.kernel.org>
X-Mailing-List: linux-ext4@vger.kernel.org

On Fri, Feb 18, 2022 at 02:20:45PM -0500, Felix Kuehling wrote:
> Am 2022-02-17 um 19:19 schrieb Jason Gunthorpe:
> > On Thu, Feb 17, 2022 at 04:12:20PM -0500, Felix Kuehling wrote:
> > 
> > > I'm thinking of a more theoretical approach: Instead of auditing all users,
> > > I'd ask, what are the invariants that a vm_normal_page should have. Then
> > > check, whether our DEVICE_COHERENT pages satisfy them. But maybe the concept
> > > of a vm_normal_page isn't defined clearly enough for that.
> > I would say the expectation is that only 'page cache and anon' pages
> > are returned - ie the first union in struct page
> > 
> > This is because the first file in your list I looked at:
> > 
> > static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
> > 				unsigned long addr, unsigned long end,
> > 				struct mm_walk *walk)
> > 
> > {
> > 		page = vm_normal_page(vma, addr, ptent);
> > [..]
> > 		if (pageout) {
> > 			if (!isolate_lru_page(page)) {
> > 
> > Uses the LRU field, so this is incompatible with all the other page
> > types.
> > 
> > One mitigation of this might be to formally make vm_normal_page() ==
> > 'pte to page cache and anon page' and add a new function that is 'pte
> > to any struct page'
> > 
> > Then go through and sort callers as appropriate.
> > 
> > The 'pte to page cache and anon page' can detect ZONE_DEVICE by
> > calling is_zone_device_page() insted of pte_devmap() and then continue
> > to return NULL. This same trick will fix GUP_fast.
> 
> Sounds good to me. What about vm_normal_page_pmd? Should we remove the
> pmd_devmap check from that function as well. I'm not even sure what a huge
> zone_device page would look like, but maybe that's a worthwhile future
> optimization for our driver.

IIRC there are other problems here as PMDs are currently wired to THPs
and not general at all..

We have huge zone_device pages, it is just any compound page of
contiguous pfns - you should be aggregating any contiguous string of
logical PFNs together into a folio for performance. If the folio is
stuffed into a PMD or not is a different question..

> I'd propose the function names vm_normal_page and vm_normal_or_device_page
> for the two functions you described. 

I wouldn't say device_page, it should be any type of page - though
device_page is the only other option ATM, AFIAK.

> current vm_normal_page with the pte_devmap check removed. vm_normal_page
> could be implemented as a wrapper around vm_normal_or_device_page, which
> just adds the !is_zone_device_page() check.

Yes

Jason