Date:   Tue, 7 Mar 2023 11:02:33 -0800
From:   Mike Kravetz <mike.kravetz@oracle.com>
To:     David Hildenbrand <david@redhat.com>
Cc:     linux-mm@kvack.org, linux-kernel@vger.kernel.org,
        Rik van Riel <riel@surriel.com>
Subject: Re: THP backed thread stacks
Message-ID: <20230307190233.GA5892@monkey>
References: <20230306235730.GA31451@monkey>
 <278ec047-4c5d-ab71-de36-094dbed4067c@redhat.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <278ec047-4c5d-ab71-de36-094dbed4067c@redhat.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?tifNjJViLPpbpECEh2BKhkx0grjBSfCqYo1U592h32HqP7S3sZyJ9ZKp7WBN?=
 =?us-ascii?Q?jbOmxjkEQDpfJKsTY3OlzZBhq0sSIGk9wttD1Guwd+wNpfdUa86qYsKtpnV/?=
 =?us-ascii?Q?lJMkWyyWBE8eKb+youKb+2oAUSwm0opXuY3DFt13hIVqZm7PuIvBszwqb/IR?=
 =?us-ascii?Q?s6bM30KlbAL8rKbJBGCRuJv54ZEt916z1rTJ9oRA4HeVQ9JrPXugXuSt4BEg?=
 =?us-ascii?Q?KJNtdb6Od2kRBI3fEg8AJPqsVZ3T+PLO7c/L2KTu2MI6H0aeC/idxh3DlrVe?=
 =?us-ascii?Q?vXob0sJ0tO/v6HXjQ5djlrASBjR2BBSAuztnQ+oyVO884R/GTzvH1KAANUME?=
 =?us-ascii?Q?uxyreoEOH1bL0ygx8NCtkcI5mJSATd8Y8cm9UgXXAefCd+mgRGVvTIuafA6O?=
 =?us-ascii?Q?73SZZKhAK6X0BeQ4ATB3GBjcS+1nQCRQwnnyoM70gmPy5eHgcwuwfuudyHfe?=
 =?us-ascii?Q?+YU8BQOUtQ+eqsDPEDRrTremTHAetkvDGBxXeSSm1+9UfjuZ5hexyUdrqHBR?=
 =?us-ascii?Q?EgQefNwbUkKA+2Mdz/T3FnGsgNaUPEb5wH46Dze6DPEQqdzE5R0+DGJtwYAe?=
 =?us-ascii?Q?lReVB8CGjoP0dCTjLIR9y3LeuGUnvBE5aj8MovU/0MuNygdbcgaV6V5ZW9g6?=
 =?us-ascii?Q?sxVVSZ3qsjTnjmW6P2wQywtFTxuH6H6Q9qt+Py4+pamwiG5iW8yUFk/iMVfa?=
 =?us-ascii?Q?VYbEVtKbxYr1EpjM4BeQzMulGC0U/VcuqbcHwRTjdgT6KZrViXMZIsnz1qYs?=
 =?us-ascii?Q?e31Fs0HDEym5EufJAyfzHhUXlrxOUmD9aEAjggY5atMoi74a5xZc6AsN+T3n?=
 =?us-ascii?Q?nPnxNSN3Xga5DnNuurk7FgRH+bLzDCL7TfHYcsbMXu4095HuO0nFy3B70yi6?=
 =?us-ascii?Q?F/kZHKUMKNiXLWkxFwE0J6DqjmItQDUnz1IoV1ZBrFYzOwOZ6PaYNv0prrzn?=
 =?us-ascii?Q?BfjVR2qPWq/cGJ2rlTz2q+jZXWCJgR0r8eawdhL1HjV3DwJVfEDr4cbh72F9?=
 =?us-ascii?Q?7fymWyebR3jzzpCx+jGcXCqb4lB5USca57u6KuNlROOa14zwjq/OKgm+kwhm?=
 =?us-ascii?Q?v43AtcBryviKS5wV5ifTF3mWAmQ89L8qv2hhC2s/AffmXSOXBeX0Xw3PzXTd?=
 =?us-ascii?Q?Osig4vssZvNQQ7dUrEohj48LKo3A+HznM9nU3RyHVQtGymxk/NqdfbPZq0s4?=
 =?us-ascii?Q?PfLIsd6ZgJpEbkgT4WPWMItzXifhuzHKb/0gdSTyAa1gcMgng8VxjthITisA?=
 =?us-ascii?Q?dkJNE2bBk2hEBLEmszZIADU7NzgA8vXBM7p8sudDB3+8DE0sMjyb9vpxkLMs?=
 =?us-ascii?Q?B2CpMuchdu+E0qFNbZWty+iq/AQLmvexO1jwzoXfLak0Cvkma/wVAjbz/jaI?=
 =?us-ascii?Q?SkUo2oollX7EgjrdCIb+WqDRsiQQPyE9Kjlu0tf3BUn2NUTVZWT8B1471mus?=
 =?us-ascii?Q?oTNzNgy7UN9j7T/ZHeuSW05s8QFy8dVHBarz+ULzGUssp+0/rdcfD8VObG8T?=
 =?us-ascii?Q?WHhrz8lhb1HwtdzB5mLW9wboNrWDEN7iQu2sTYfiuU+Iui3yi22Bq8R3PQ2F?=
 =?us-ascii?Q?k5jfJXWTrx7QeUWqHJBIRX9nrOTeLSe9oT1dEzF5Oz+Ya3q91sjG5UqvAFl1?=
 =?us-ascii?Q?vg=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: aueEezSQbF4iGf9hohUxcRMS50QxCp0FNQ/xoFJBi4D0q/iYzzkFkeHJ59C5QA+XK0FFSMA+uSa2s3HB2Tkcl8CekyEz1UHra36PEDMQ09sevB1iby9+u4EA1BPfSkaHuADAKYLgc7vqPuGHjOiXrK8YHqgXggBX6/1jKWG5yG9SrnHtv79tH+HtHtI53aAJAHi7/dN5rBeXkHlxKDyIWZIEz3foO0B7+zJ1M0Uo/kG5vVh3oPJ813t9/KM3ditI/OMKqOr7OXBbISI5cbYLcGxSaTjIwxl30JBLVRUfXzBTo39oh5ZsOV0p5TFxKC8Qybx05bFpQnYG1UuFkunW0aGvm7zQqWfGFZpiByJLdRNrl1IiWa8shtI8vL3eB6z84lK/Pty80/c4feIrDFklgLBft4AjS+mqj9KMHa6q0+Wc6E00Ox8JXM81No8v5AbcIEKhLwUdy1+McQVm21QjnHnjtwg+PK5ylbI36OhJORlipXwcQ+CiyH6KCIY9jjJsW9cRNakfjzWfOig45SBohkTyGrkVrnzWA404bfFdjdmYoNX6hmMWH9bdZt+z2AcTGnvolGMrQUpee9XKxOZSdW5/lSFTs/vfxsInvcGv4qgaxqoeOJOUbrRs6X2Yi2rtP9VolfWdBTUBfify2sZroM74JjHpdre3wZCPlplWLOU0XRX1Se0a+FyH1RnaMWK4org+QXFN5s/kSnmP7PCcnr4Lrf6T7IV8DVJk4A4QV3Eq/negKk94VRQ77ka2sWQnRwHcBpX5P/eL2ciz5iaZDHL+5o4tn8AXJsO2xWg/4VkK3TQiRTS+721RWHdY2vOy68Aum58gG42nvYfX6zjoMwAD4SF5G1AiEQ9oZYHMsQU=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c70a7c0f-aa11-45cc-f797-08db1f3e83be
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Mar 2023 19:02:36.3227
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 0fbBNvr86p3jlkr4wBjpVzF8JbrpknPUWlh8EtI3ZAC5N8R1dh7hcrgm2lmShPG6qByZzW+UdHeJNjWmYysnJQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW5PR10MB5667
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.219,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22
 definitions=2023-03-07_14,2023-03-07_01,2023-02-09_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 bulkscore=0
 malwarescore=0 spamscore=0 mlxscore=0 suspectscore=0 adultscore=0
 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2212070000 definitions=main-2303070168
X-Proofpoint-GUID: bVAZjYXwzEHLJiPUIeaGPEFCZLztJtsA
X-Proofpoint-ORIG-GUID: bVAZjYXwzEHLJiPUIeaGPEFCZLztJtsA
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 03/07/23 11:10, David Hildenbrand wrote:
> On 07.03.23 00:57, Mike Kravetz wrote:
> > One of our product teams recently experienced 'memory bloat' in their
> > environment.  The application in this environment is the JVM which
> > creates hundreds of threads.  Threads are ultimately created via
> > pthread_create which also creates the thread stacks.  pthread attributes
> > are modified so that stacks are 2MB in size.  It just so happens that
> > due to allocation patterns, all their stacks are at 2MB boundaries.
> 
> Is this also related to a recent change, where we try to always align at PMD
> boundaries now, such that this gets more likely?

Nope, it happens on a kernel without this change.

> commit f35b5d7d676e59e401690b678cd3cfec5e785c23
> Author: Rik van Riel <riel@surriel.com>
> Date:   Tue Aug 9 14:24:57 2022 -0400
> 
>     mm: align larger anonymous mappings on THP boundaries
> 
> 
> As a side note, I even heard of complains about memory bloat when switching
> from 4k -> 64k page size with many threads ...

It seems like the 'answer' is to have applications explicitly opt out of
THP if they know it is detrimental for some reason.  In this case, it
makes sense to opt out for thread stacks that are known to be 2MB at most.
Unfortunately, this means the application would need to replicate stack
creation (including guard pages) as well as cleanup that is done in
libpthread/glibc.
-- 
Mike Kravetz