Received-SPF: pass (google.com: domain of linux-kernel+bounces-121659-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Date: Wed, 27 Mar 2024 13:57:54 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: Christophe Leroy <christophe.leroy@csgroup.eu>
Cc: Andrew Morton <akpm@linux-foundation.org>, Peter Xu <peterx@redhat.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-mm@kvack.org" <linux-mm@kvack.org>,
	"linuxppc-dev@lists.ozlabs.org" <linuxppc-dev@lists.ozlabs.org>
Subject: Re: [RFC PATCH 1/8] mm: Provide pagesize to pmd_populate()
Message-ID: <20240327165754.GM946323@nvidia.com>
References: <cover.1711377230.git.christophe.leroy@csgroup.eu>
 <54d78f1b7e7f1c671e40b7c0c637380bcb834326.1711377230.git.christophe.leroy@csgroup.eu>
 <20240325161919.GD6245@nvidia.com>
 <6e16e042-0143-4a52-b4b7-09cf0022bc3e@csgroup.eu>
 <20240326150118.GI6245@nvidia.com>
 <9703878c-c0b0-48ff-a356-d43e8f7391f3@csgroup.eu>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <9703878c-c0b0-48ff-a356-d43e8f7391f3@csgroup.eu>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?UFcX41rQNJOxi+P7L3uCouOW7O1GOq8HC0kaCjvUs9zQadcbWeTlIXtlOQAF?=
 =?us-ascii?Q?PVk+iewfjMw2nxI09UicHku2ORH3FvEeW42hn8hEG2F4QcZxItYuWJQuorYk?=
 =?us-ascii?Q?ascN4jmg8xjxF5rX65mT7a7Yz0ietLKD+2zRi/mt8R8aBmdFiBszi1qdO24m?=
 =?us-ascii?Q?vyBbcTpC5KPWNwNQ7tEzFCxRrVdhH0pv8i979a8v7Mwtu5Za7y0MTzwACUb+?=
 =?us-ascii?Q?e90+rNvqxwhFBI3ot7MMOCnv1QIHXugiX+0NNWEZDp/ARshhkw9SdJj2CT0J?=
 =?us-ascii?Q?P3pCTnu+gHDkcw84BtwM5iOjsTMb/o9hdbt3Kraxq5pnNNhOKX/wKfVOu7FD?=
 =?us-ascii?Q?iookODuCL1VMGjpEQwsw81jcdMVH3Zo1ORHHotHpDpczljNX8ehcBA4gdogf?=
 =?us-ascii?Q?4tTLF2A9K2+a+ycBRaizcUPIMy1DIE6gGJTU/sKsBq5g8jxqjFl6KxcPzL7H?=
 =?us-ascii?Q?SbJBGFgBSnO0+NIeFQ6YOrA8JLevkUSf76BTaWu6XotmNPxy2NTyKy6fXdIb?=
 =?us-ascii?Q?yQZ5ePacR4nOEVfHR+wMXD6kb8AMggm1zkYvfOmlt3mPArmo25nfQA7o1Nla?=
 =?us-ascii?Q?VnAPYLDZm/5Oaq4sJCbpB/M91rrLK/QOIuL0JLH6I9+0hJ3+KWWG0zFJF3oB?=
 =?us-ascii?Q?jeOdJ7Opn35wyu1Pe0PfqCnI2XOJskpGDvf/1K1f/h1qHeaonn+qTaSRLb8d?=
 =?us-ascii?Q?/Z3czsM8zMlLR4UMiRbEgblP5yprauQFoEfDUSweE1C64la/LS0oqVAeDWGO?=
 =?us-ascii?Q?NNQfjFn/AAcXp++/A2J3J4NRvh7eI+taZkLlLlJ8cuYgxPtgu+/IDkd04pMC?=
 =?us-ascii?Q?94PkTbT4gB4eAucbPrqFATHYIYKK7CBbgwjnS1T/6ZU2w6h/PtOVf5ig2tmC?=
 =?us-ascii?Q?aSrUvHve4t3A7Su4HMOtuVANBMuObI0Z3OfTGeynM/NOXeJVGQNaSAfNloWo?=
 =?us-ascii?Q?y949y+NGt8mtnTg1v+qTsGcB8Pm5IqayKqdgYis9be0ctngb6bU0EDl0L3Hh?=
 =?us-ascii?Q?slpDKuUMMp0PHvt/jXeR3GJeVW9/A+Q7BUrx84ln1FP3eu8YaYL+da0nWA+w?=
 =?us-ascii?Q?q5gSu+1N/zgNMX6a8B+6LkBbg3BK9TBISsBaXAAn5Y1DB+rydqW1Bmp1C7mO?=
 =?us-ascii?Q?DL7oXjYdsCsVjjwqVL81+k3JS2NiApG9Rf965akg97P/Wl2RSYmeIC51NBH5?=
 =?us-ascii?Q?OXjlseA6GB0/02w807SQiQA6hS3Gbq+I/ASQtCrC/P4hJJjDdG286cCV8Eng?=
 =?us-ascii?Q?uws76d+9U0M8uX1EKayXnfRkGQ+xqBlOFyvBSEve3H2dUobex1zNiU5PYxAD?=
 =?us-ascii?Q?HLMB5uqzF08FTJCZC+8gid3ZX+10N+xRjm23t6uXtQHbmjXBu4OXMCKJVceA?=
 =?us-ascii?Q?Dv6NcEROKi9kNOadSZcv5lXWEfLetT5u5rjO/XrbE+t6R9a8fytJi8R6y9TI?=
 =?us-ascii?Q?Aia9R6ufsJMvpjUEAJlJ5RnR7WHEkU8hFl+ecONgv1i09TbdMpfrmOPhjYN7?=
 =?us-ascii?Q?57gDT1w91vi7slokJYdtrPusdQ+pPJqGjUqs+bDx0zsLEkWLeEWQjsrTGTy6?=
 =?us-ascii?Q?CeR6pcT/Qt3Pv3tZgb7Kyc6uGiJteqWgphC8CMDu?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4168e448-ebac-4e51-c2b3-08dc4e7f0c1d
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 16:57:55.1915
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 3/QQTHNKIUIFS5bJVUKR0YeWZHtqKJ0dTkn3aXZv6CBjUV/Joibr3x9+J/TSxbj/
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4423

On Wed, Mar 27, 2024 at 09:58:35AM +0000, Christophe Leroy wrote:
> > Just general remarks on the ones with huge pages:
> > 
> >   hash 64k and hugepage 16M/16G
> >   radix 64k/radix hugepage 2M/1G
> >   radix 4k/radix hugepage 2M/1G
> >   nohash 32
> >    - I think this is just a normal x86 like scheme? PMD/PUD can be a
> >      leaf with the same size as a next level table.
> > 
> >      Do any of these cases need to know the higher level to parse the
> >      lower? eg is there a 2M bit in the PUD indicating that the PMD
> >      is a table of 2M leafs or does each PMD entry have a bit
> >      indicating it is a leaf?
> 
> For hash and radix there is a bit that tells it is leaf (_PAGE_PTE)
> 
> For nohash32/e500 I think the drawing is not full right, there is a huge 
> page directory (hugepd) with a single entry. I think it should be 
> possible to change it to a leaf entry, it seems we have bit _PAGE_SW1 
> available in the PTE.

It sounds to me like PPC breaks down into only a couple fundamental
behaviors
 - x86 like leaf in many page levels. Use the pgd/pud/pmd_leaf() and
   related to implement it
 - ARM like contig PTE within a single page table level. Use the
   contig sutff to implement it
 - Contig PTE across two page table levels with a bit in the
   PMD. Needs new support like you showed
 - Page table levels with a variable page size. Ie a PUD can point to
   a directory of 8 pages or 512 pages of different size. Probbaly
   needs some new core support, but I think your changes to the
   *_offset go a long way already.

> > 
> >   hash 4k and hugepage 16M/16G
> >   nohash 64
> >    - How does this work? I guess since 8xx explicitly calls out
> >      consecutive this is actually the pgd can point to 512 256M
> >      entries or 8 16G entries? Ie the table size at each level is
> >      varable? Or is it the same and the table size is still 512 and
> >      each 16G entry is replicated 64 times?
> 
> For those it is using the huge page directory (hugepd) which can be 
> hooked at any level and is a directory of huge pages on its own. There 
> is no consecutive entries involved here I think, allthough I'm not 
> completely sure.
> 
> For hash4k I'm not sure how it works, this was changed by commit 
> e2b3d202d1db ("powerpc: Switch 16GB and 16MB explicit hugepages to a 
> different page table format")
> 
> For the nohash/64, a PGD entry points either to a regular PUD directory 
> or to a HUGEPD directory. The size of the HUGEPD directory is encoded in 
> the 6 lower bits of the PGD entry.

If it is a software walker there might be value in just aligning to
the contig pte scheme in all levels and forgetting about the variable
size page table levels. That quarter page stuff is a PITA to manage
the memory allocation for on PPC anyhow..

Jason