Date:   Mon, 20 Mar 2023 11:06:52 -0700
From:   Mike Kravetz <mike.kravetz@oracle.com>
To:     William Kucharski <william.kucharski@oracle.com>
Cc:     David Hildenbrand <david@redhat.com>,
        Matthew Wilcox <willy@infradead.org>,
        Linux-MM <linux-mm@kvack.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: THP backed thread stacks
Message-ID: <20230320180652.GA4127@monkey>
References: <20230306235730.GA31451@monkey>
 <ZBSo+mLUOsKvy3rC@casper.infradead.org>
 <20230317184632.GA69459@monkey>
 <655ca9fd-ddc3-0bfa-b442-fc60cda02baa@redhat.com>
 <4A61B05E-62C6-4BC1-9301-CF12FC5A252F@oracle.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <4A61B05E-62C6-4BC1-9301-CF12FC5A252F@oracle.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Ah5pFzimyIeQ9QXca89FwrRV3WcVvOZqWPJAUAuULMImgZsoAIJeUDKeuj5b?=
 =?us-ascii?Q?yNf6jeVnarIz4J3FdPGF2Gz9xrZrLMqxB4arP1/ymxroLyil+bqtdXVkX0QU?=
 =?us-ascii?Q?u9ijvrlj9bAsQje9WvW7R9RGMdFx8zJEasdACkpXAWroec6gv5FzhwkAGR99?=
 =?us-ascii?Q?ezrOAUcjKg4uYZ++dccg2tpbYd02H7sv2Joy+8oZfl4c4BzRP7JQ/zPu4jCJ?=
 =?us-ascii?Q?0URKLLkMw3+yArU4bNUAQ6Qu3axAbMMeXbdEgLAH/jNof2aE6PUhRl1Qz4aP?=
 =?us-ascii?Q?uBpl5r7EIT/Mj2620mjilsXuxICOnw3PZ2EBVQ9vvapvAv7pyFebGBiimOL7?=
 =?us-ascii?Q?CozedcqSpP08rZgsPakeKlpoL/V8AH673DQuPz8izjx/YXUPbvY/ZBCO+zmD?=
 =?us-ascii?Q?lAgddy9bhOvXgsr84snhwDzJmrX2gDV7m2SnSjAJBVshF75iae5j3d8hLZ9G?=
 =?us-ascii?Q?L2Ct/AkMYq2mEnHgtoOG8Hn8HsKsK5SN6/Z9mxmvbjsKAQBAMByGpgdjz3LC?=
 =?us-ascii?Q?W79t/H/AOjsQbhvBh21D5a2h5TwT4g6xhq4uULTq777araEeliaZz0uaMS6I?=
 =?us-ascii?Q?YANAkbaCOVi4Bgp0iy58jRQ9RTxe6oaIjd4yogDuIDndEpvfhYgAi8usaqko?=
 =?us-ascii?Q?XPPz7zCBi3bNC/Y6wnxQeO6zk5JOqrSYwtZ3z7UV2Eyz6GcvCl9vn/iSmZrp?=
 =?us-ascii?Q?ojnkPSAjsK8a3s1GrrQdL7YynOtMPfteR49QfwjoMl8KBwNz2s46CBCz0774?=
 =?us-ascii?Q?1f6N5RgvoqjZzIiNBCszGcnv5DL000haTLOgc5uG3AXy/4RX1GEthIGZpoEn?=
 =?us-ascii?Q?8I+H0atBUsso5kXgw7S1bTaSSkyA67S71zGBvbI2kq2+1X7lpiFW64O8eNJ1?=
 =?us-ascii?Q?SCfORU4UOmlgVJOKus/b1ng84Bf/2ZfnG8nsAmVw7KdVw0JUaZUMMNwvgsRH?=
 =?us-ascii?Q?oYmwTNnAs7IKtCU6F+dHGM1xuVyKBCrSmSmEZREKgBbP+VmlGm/FsdqcnQ/k?=
 =?us-ascii?Q?cfDuHf+xBzlLAiQUlHHHh9PERd5rE6bUFBapWq/4y9lB0YPuVRICiyb5ut2F?=
 =?us-ascii?Q?XaPZRR/zQMEHjnpTwfBSs3MDIb2DJTmwkBsl0/5z7m5axf/mxXknhmebB7Mz?=
 =?us-ascii?Q?vcsbqHD2C6UQr1ox5uI4NUGmqgNEqzqud+8FAGY2byg4fD4doFkCrc1u9PjH?=
 =?us-ascii?Q?MWY4/OU8ruoG4XngZd6foyWrKvjeae3Oyxg36nunYs7oUW3Sp9Khb6pUizZX?=
 =?us-ascii?Q?EOipLzAD7L9MzplCNqCrVS1rahKiYHlXWyPWNKfx21vQWJcZeAuLH+AvVMFo?=
 =?us-ascii?Q?bi0GBLCFsOXLzsaFhtpqdi2D+AWL0ABecvdhL/La78dFEcY+U5IeyTQJW7n/?=
 =?us-ascii?Q?7yI44pDUsVd0EcRur/GOSaig8cFEIH4ZbXAlHWz2LbJ17CPUovXPj7Qhzkeb?=
 =?us-ascii?Q?X4uDwhLAjT/MmEntLc55gj0NesS/AUrht0c7+UsPfxR3JBGTM4Lo139akPpN?=
 =?us-ascii?Q?4roFsLSaw6dscLtdFLq51ckaB7fpleQWXVUodfCQv/Q2ZL/f8jwFEF79C6Pw?=
 =?us-ascii?Q?bNDshaGUscSW5TutDNVTezZf3o36X7Vd0zrtD730JSGCd5dL75blxw6b4X+4?=
 =?us-ascii?Q?bA=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: pSwvA8ocQfH0f5MeGlXxAHS5NKcN9prseKt2uhKSUpYAzv1biC28xpnZTNp6YZ75fEmysSRuLQa70dPJTAWcjFVz7n4mCNA5QqRbcERPBsuA/HQi/mkRSdDfYbaCQmHTK8yAhWiUnnsHKiqowHR7uGijoYKLkdfE+PeOIMvPSIjAtMN3gJfSdChP4UpYyP9dbKx4EtI8TdcjRacX0V87R4cVciKS1cBy3z1mAv8jScZhNVAxtNRpZ2NM1ra9LWUOtb+k32pdC6sLJ2jxo9Tcs42LWqdPMCNKwCjW8KAKlxVb6qAEbjS+SieWYf5lqhDuVXPj1VcJBPTKIgh7ut6B4coAYB0cyA9PZ+5SgQLf8BloceOk+MxMwoiCBnrwhDL+83yc4lnRxBBjGlsm3oZA3dQMefGpM6ArnYD9V0i6cKug9ZPidnVeNEm56B2KYJBZPms5vgTouuikzkwMffZ644j07/w6rPRowLQShtQMyll0cn/LDiuqnJV6JynV+mMJdUQPsZ2Cfhvs5Qf+UsI4jVefNqVVUG22Ia0nu5d3XdLQ7ww4XOZmYKiLWuew6dhFN9JeC1aUhTS3iZNCgtntxSxlEInoiTZh3Hn5Rs845LiWsXblts2qZxtcMt2xCjmbLGlz+oEBXBk1Y2eDFzMxvJfhBAF3hiqsRodLKWb2ZCE0L32afqALOJXLKworXK+URAvUL8Q54gDczTNzuMDcik7S3OMUwvlJld5MPGKgxO1r5RL4qENfDWzpRzDtWhMwhZhDybgNtBX5uAwCZFLG6aCVugYWdQ2XyHfJ58qG/Y7jtSLFMFAjBUptb89MhZSjvNdl8fa3uJJh9YcQpzO2NNgzWzk4wLc0mImoFk6qy5A=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c504c3b3-62bd-4c71-b5bd-08db296de172
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Mar 2023 18:06:51.6903
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ckqTjxatqj0rQKSuGb/0c/KkUTaOJVeFKc+T3/D+wzckPXWFBTEDlZkyKJQcp9ahOmn/gO/2K5bcfko4bwkBzA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR10MB6883
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22
 definitions=2023-03-20_15,2023-03-20_02,2023-02-09_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 spamscore=0
 suspectscore=0 mlxlogscore=999 mlxscore=0 adultscore=0 bulkscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303150002
 definitions=main-2303200152
X-Proofpoint-GUID: smqL96FuBH3Vd4qBosr0b4SzItrpEn9B
X-Proofpoint-ORIG-GUID: smqL96FuBH3Vd4qBosr0b4SzItrpEn9B
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 03/20/23 10:46, William Kucharski wrote:
> 
> 
> > On Mar 20, 2023, at 05:12, David Hildenbrand <david@redhat.com> wrote:
> > 
> > On 17.03.23 19:46, Mike Kravetz wrote:
> >> On 03/17/23 17:52, Matthew Wilcox wrote:
> >>> On Mon, Mar 06, 2023 at 03:57:30PM -0800, Mike Kravetz wrote:
> >>>> One of our product teams recently experienced 'memory bloat' in their
> >>>> environment.  The application in this environment is the JVM which
> >>>> creates hundreds of threads.  Threads are ultimately created via
> >>>> pthread_create which also creates the thread stacks.  pthread attributes
> >>>> are modified so that stacks are 2MB in size.  It just so happens that
> >>>> due to allocation patterns, all their stacks are at 2MB boundaries.  The
> >>>> system has THP always set, so a huge page is allocated at the first
> >>>> (write) fault when libpthread initializes the stack.
> >>> 
> >>> Do you happen to have an strace (or similar) so we can understand what
> >>> the application is doing?
> >>> 
> >>> My understanding is that for a normal app (like, say, 'cat'), we'll
> >>> allow up to an 8MB stack, but we only create a VMA that is 4kB in size
> >>> and set the VM_GROWSDOWN flag on it (to allow it to magically grow).
> >>> Therefore we won't create a 2MB page because the VMA is too small.
> >>> 
> >>> It sounds like the pthread library is maybe creating a 2MB stack as
> >>> a 2MB VMA, and that's why we're seeing this behaviour?
> >> Yes, pthread stacks create a VMA equal to stack size which is different
> >> than 'main thread' stack.  The 2MB size for pthread stacks created by
> >> JVM is actually them explicitly requesting the size (8MB default).
> >> We have a good understanding of what is happening.  Behavior actually
> >> changed a bit with glibc versions in OL7 vs OL8.  Do note that THP usage
> >> is somewhat out of the control of an application IF they rely on
> >> glibc/pthread to allocate stacks.  Only way for application to make sure
> >> pthread stacks do not use THP would be for them to allocate themselves.
> >> Then, they would need to set up the guard page themselves.  They would
> >> also need to monitor the status of all threads to determine when stacks
> >> could be deleted.  A bunch of extra code that glibc/pthread already does
> >> for free.
> >> Oracle glibc team is also involved, and it 'looks' like they may have
> >> upstream buy in to add a flag to explicitly enable or disable hugepages
> >> on pthread stacks.
> >> It seems like concensus from mm community is that we should not
> >> treat stacks any differently than any other mappings WRT THP.  That is
> >> OK, just wanted to throw it out there.
> > 
> > I wonder if this might we one of the cases where we don't want to allocate a THP on first access to fill holes we don't know if they are all going to get used. But we might want to let khugepaged place a THP if all PTEs are already populated. Hm.
> > 
> > -- 
> > Thanks,
> > 
> > David / dhildenb
> 
> Unless we do decide to start honoring MAP_STACK, we would be setting an interesting precedent here in that stacks would be the only THP allocation that would be denied a large page until it first proved it was actually going to use all the individual PAGESIZE pages comprising one. Should mapping a text page using a THP be likewise deferred until each PAGESIZE page comprising it had been accessed?
> 
> Given the main questions of:
> 
> 1) How to know whether it's a stack allocation
> 
> 2) How to determine whether the app is consciously trying to allocate the stack via a THP or if it just happened to win the address alignment/size lottery
> 
> 3) Whether to honor the THP allocation in either case
> 
> It seems taking the khugepaged approach would require Yet Another Flag to provide a way for an application that KNOWS a THP-mapped stack would be useful to get it without having to incorporate a loop to touch a byte in every PAGESIZE page in their allocated aligned stack and hope it gets its upgrade.
> 


Just another 2 cents thrown into the pool.

We currently treat the 'main thread' stack differently than pthread stacks.
My understanding of main stack handling matches that described by Matthew
above.  So, IIUC the only way the 'main stack' could possibly use THP pages
is if base pages were collapsed via khugepaged.
-- 
Mike Kravetz