Received-SPF: pass (google.com: domain of linux-kernel+bounces-29018-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Date: Wed, 17 Jan 2024 09:22:43 -0400
From: Jason Gunthorpe <jgg@nvidia.com>
To: Christophe Leroy <christophe.leroy@csgroup.eu>,
	Ryan Roberts <ryan.roberts@arm.com>
Cc: "peterx@redhat.com" <peterx@redhat.com>,
	"linux-mm@kvack.org" <linux-mm@kvack.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	James Houghton <jthoughton@google.com>,
	David Hildenbrand <david@redhat.com>,
	"Kirill A . Shutemov" <kirill@shutemov.name>,
	Yang Shi <shy828301@gmail.com>,
	"linux-riscv@lists.infradead.org" <linux-riscv@lists.infradead.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	"Aneesh Kumar K . V" <aneesh.kumar@kernel.org>,
	Rik van Riel <riel@surriel.com>,
	Andrea Arcangeli <aarcange@redhat.com>,
	Axel Rasmussen <axelrasmussen@google.com>,
	Mike Rapoport <rppt@kernel.org>, John Hubbard <jhubbard@nvidia.com>,
	Vlastimil Babka <vbabka@suse.cz>,
	Michael Ellerman <mpe@ellerman.id.au>,
	Andrew Jones <andrew.jones@linux.dev>,
	"linuxppc-dev@lists.ozlabs.org" <linuxppc-dev@lists.ozlabs.org>,
	Mike Kravetz <mike.kravetz@oracle.com>,
	Muchun Song <muchun.song@linux.dev>,
	"linux-arm-kernel@lists.infradead.org" <linux-arm-kernel@lists.infradead.org>,
	Christoph Hellwig <hch@infradead.org>,
	Lorenzo Stoakes <lstoakes@gmail.com>,
	Matthew Wilcox <willy@infradead.org>
Subject: Re: [PATCH v2 06/13] mm/gup: Drop folio_fast_pin_allowed() in hugepd
 processing
Message-ID: <20240117132243.GG734935@nvidia.com>
References: <20240103091423.400294-1-peterx@redhat.com>
 <20240103091423.400294-7-peterx@redhat.com>
 <20240115183748.GR734935@nvidia.com>
 <c60c9d88-33aa-4312-a23c-20206e503b6e@csgroup.eu>
 <20240116123138.GZ734935@nvidia.com>
 <44e450cb-5d3f-407e-97a3-024eb936f74b@csgroup.eu>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <44e450cb-5d3f-407e-97a3-024eb936f74b@csgroup.eu>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?UKoIFMwIgo+UP7YkCIUpJly78vCqIGGehMSOnenDhHo9xvv60z5FvHyRuyLg?=
 =?us-ascii?Q?uHqPA67mf1TDwygj+ayP55jpqpmFdhCv8VI6pPwDM4GuwuSAb0laEi10hPPC?=
 =?us-ascii?Q?batia6+j//Wl5HtR8yxc32miN2LpnzYn8EzT9x2jXe0TWiXl4eQp9ZXqg9dW?=
 =?us-ascii?Q?o5gC2i05Rf1HPwP8M5j4eNvGiQDmYtBR2Se6i0L77YHom4MsmV+B6NHkP4kc?=
 =?us-ascii?Q?AAuUvT+Z95VCfaHM3WwCIzEEIKZfp7slujsD6iDccCWXr5kJPZgguMamI3qC?=
 =?us-ascii?Q?2QajsFnpbEjGazpMLcKaZHFRKvynBc0+cY/PV6LMbzkD/iHCELC07TztNxjY?=
 =?us-ascii?Q?YDrybwwqo7DzN58FKLKghN7Z9BnzRxAmIs8tkgXz+YYshm9o6h4uq8GOORxZ?=
 =?us-ascii?Q?D7v4f88dxvKCleB3Cxo1jZvqZtawQcpJAHlmD5Yc3zD7L46gOeVlHJkiXPbK?=
 =?us-ascii?Q?D3UKK/XK+eclMeLAzDA6WbZ4ypbIPAQKNvYMyPKCT2hHT6WvSP4mIIkT1w1F?=
 =?us-ascii?Q?F0U4Vd0ZlOCsjkBBZnKZnb0XwRTSXY7yWnoMhbv3g1vq8tpgscI391PMsdab?=
 =?us-ascii?Q?x+kH5jJJlxT5uqByKMgDKk1JPeQuuBZCSx+34H+Skm9reES2tiB/LGKXlrf5?=
 =?us-ascii?Q?bEaM1HstDiu+GPxfN8t+P303I4Ty8nMK7Axcn6g6NCDaEx5v1L1NfLRSk/Gc?=
 =?us-ascii?Q?P23hqXHSTh517xlf3MB0rzIjXdQDNta7jWNS1L1izNb1Xd7e88YY+403v7zx?=
 =?us-ascii?Q?tCqMa5LTK/B2V2d9o6OtP2gaZG2VTIbH1EiQ/TME2zp7TfDCOBi9WS+HTB6K?=
 =?us-ascii?Q?+itNLJQWgtTCVyPaYsyZexsD8mKXy/WVZZGrmszFkxCd0siSs1q5i53rgwCY?=
 =?us-ascii?Q?nZb8XyjO49GgthVWBSNhz9xSkZsZ/mhXLeQgxsocFLlchYRlYf0U8PVc5Hc2?=
 =?us-ascii?Q?B7wNFA6J2nwtg8mQf7ORorj4bN+T8JiQ1o7mVKJ+XrXPrIp05g7BaMZxGWjj?=
 =?us-ascii?Q?jXcREvSWROFWCdoeh6wx86N+AuCDCN0KKB+gD8KgO2l3hpeyKrOtog1zZ8eP?=
 =?us-ascii?Q?/JricatpSrCtMNE98BvsWUX2BiqXeteeQi2GZR1JzsKsygCalWTgpNi48gaK?=
 =?us-ascii?Q?QFTuXDHabTDyzNfwbo1BroVpxMHZw1b+1PhQ+6CnxS2S2xXrBWFtPQBbew7T?=
 =?us-ascii?Q?rgijQRgTVd0rF8W9686GLw+Cuvw5SOxKnra9Mf0E17mrXRyxTColzFkPeb1F?=
 =?us-ascii?Q?MnudxA7zpNtCQn6+1vYYdjBQ/jNDNU/yY8SgVfWMpKwFzOA1wt52BoAq0KVf?=
 =?us-ascii?Q?GNb2SAhY43kCqmdrHE8QIu0e7KOOfcHesuKrClRB7mmHkUgw7OtFaF+cKZq2?=
 =?us-ascii?Q?zXs94xWwRCSK1QgWuKDZ55WqXH+l2h+6ETJYqz0lRnLAvyhIq0QYchCwadpt?=
 =?us-ascii?Q?OHe2UaH7TO1VH/fi6nptLusm0ZQL/savXJgpZZoDzMCV21LyvUMf6IQ22fUc?=
 =?us-ascii?Q?cYXbmitxRp5efeL0gbfH+VbSjw5xmmnaaQYO+x+xWGpqehokSuOEiJ8xs7wI?=
 =?us-ascii?Q?MyKZ9mEg6fWhtTtn3xaEh5TyrmdECxwdOfSB02ix?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a90389c4-06a8-435f-bdd5-08dc175f63db
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jan 2024 13:22:44.5643
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: YuRr4k9IImmVkC4BHJkkI+6OMg5e5p1r5rksakFXdof9PPb19nQO7yfpICDwcwtl
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8285

On Tue, Jan 16, 2024 at 06:32:32PM +0000, Christophe Leroy wrote:
> >> hugepd is a page directory dedicated to huge pages, where you have huge
> >> pages listed instead of regular pages. For instance, on powerpc 32 with
> >> each PGD entries covering 4Mbytes, a regular page table has 1024 PTEs. A
> >> hugepd for 512k is a page table with 8 entries.
> >>
> >> And for 8Mbytes entries, the hugepd is a page table with only one entry.
> >> And 2 consecutive PGS entries will point to the same hugepd to cover the
> >> entire 8Mbytes.
> > 
> > That still sounds alot like the ARM thing - except ARM replicates the
> > entry, you also said PPC relicates the entry like ARM to get to the
> > 8M?
> 
> Is it like ARM ? Not sure. The PTE is not in the PGD it must be in a L2 
> directory, even for 8M.

Your diagram looks almost exactly like ARM to me.

The key thing is that the address for the L2 Table is *always* formed as:

   L2 Table Base << 12 + L2 Index << 2 + 00

Then the L2 Descriptor must contains bits indicating the page
size. The L2 Descriptor is replicated to every 4k entry that the page
size covers.

The only difference I see is the 8M case which has a page size greater
than a single L1 entry.

> Yes that's how it works on powerpc. For 8xx we used to do that for both 
> 8M and 512k pages. Now for 512k pages we do kind of like ARM (which 
> means replicating the entry 128 times) as that's needed to allow mixing 
> different page sizes for a given PGD entry.

Right, you want to have granular page sizes or it becomes unusable in
the general case
 
> But for 8M pages that would mean replicating the entry 2048 times. 
> That's a bit too much isn't it ?

Indeed, de-duplicating the L2 Table is a neat optimization.

> > So if you imagine a pmd_leaf(), pmd_leaf_size() and a pte_leaf_size()
> > that would return enough information for both.
> 
> pmd_leaf() ? Unless I'm missing something I can't do leaf at PMD (PGD) 
> level. It must be a two-level process even for pages bigger than a PMD 
> entry.

Right, this is the normal THP/hugetlb situation on x86/etc. It
wouldn't apply here since it seems the HW doesn't have a bit in the L1
descriptor to indicate leaf.

Instead for PPC this hugepd stuff should start to follow Ryan's
generic work for ARM contig:

https://lore.kernel.org/all/20231218105100.172635-1-ryan.roberts@arm.com/

Specifically the arch implementation:

https://lore.kernel.org/linux-mm/20231218105100.172635-15-ryan.roberts@arm.com/

Ie the arch should ultimately wire up the replication and variable
page size bits within its implementation of set_ptes(). set_ptes()s
gets a contiguous run of address and should install it with maximum
use of the variable page sizes. The core code will start to call
set_ptes() in more cases as Ryan gets along his project.

For the purposes of GUP, where are are today and where we are going,
it would be much better to not have a special PPC specific "hugepd"
parser. Just process each of the 4k replicates one by one like ARM is
starting with.

The arch would still have to return the correct page address from
pte_phys() which I think Ryan is doing by having the replicates encode
the full 4k based address in each entry. The HW will ignore those low
bits and pte_phys() then works properly. This would work for PPC as
well, excluding the 8M optimization.

Going forward I'd expect to see some pte_page_size() that returns the
size bits and GUP can have logic to skip reading replicates.

The advantage of all this is that it stops making the feature special
and the work Ryan is doing to generically push larger folios into
set_ptes will become usable on these PPC platforms as well. And we can
kill the PPC specific hugepd.

Jason