Date:   Wed, 8 Mar 2023 11:02:06 -0800
From:   Mike Kravetz <mike.kravetz@oracle.com>
To:     Peter Xu <peterx@redhat.com>, David Hildenbrand <david@redhat.com>,
        Rik van Riel <riel@surriel.com>,
        Mike Rapoport <rppt@kernel.org>
Cc:     linux-mm@kvack.org, linux-kernel@vger.kernel.org
Subject: Re: THP backed thread stacks
Message-ID: <20230308190206.GA4005@monkey>
References: <20230306235730.GA31451@monkey>
 <ZAaCISgq4A/GnkCk@x1n>
 <20230307004049.GC4956@monkey>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20230307004049.GC4956@monkey>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?cCafmdJNGIbU2ZFI2h67X8gSiCw7BtMwnPvzLmRLc8kbuNGb350fTc+vrLZ9?=
 =?us-ascii?Q?XoNpOS/FI5lScOZCyjH+H6lCo3NyaX4QhB7n6sKXWrOMaZERxFeSYFlBxSwQ?=
 =?us-ascii?Q?XYQM9PpWrfmjsnJGTtkmgg1DJY8ZOUtaHmrAnExI6VnuM79fO5TXSEPdOdon?=
 =?us-ascii?Q?bY00dfnRF6LSxXTsNYoLHCritsJP1yLbugEP7GTXneRcY4LhSZVfHkgSyoWp?=
 =?us-ascii?Q?8q38DPMMF26ew2q6oXH78fFSl0NBE5apm6Vb690OPwS3eRXgHGk+2ZVsRmnW?=
 =?us-ascii?Q?IWI4ax6+4/dEDo1JD6SPwSMG3dWM8Is2e0G8B9SqnI0TUyoS1vYvX1mLFi47?=
 =?us-ascii?Q?e5Zy657Fb6zEfQtV5+oe904/LmYk8TmTS5SgfTlTfPwej5lpbVe84PFFYNYh?=
 =?us-ascii?Q?1eqnKoRT2KA8XVm1FqJMcL2I2YwaUTXtmjqyx4HvtgYXwLbbVgxmde41Xbws?=
 =?us-ascii?Q?XTZWczGEW9HpyJGI2BxLURBz/g9P0mTtKERV9ku4qVkvoyy6HVln1H9H7Jfm?=
 =?us-ascii?Q?z9q6xTSKAbuyZqd8/YFjNmN1zTpWF491Xqp7Eij8DBC5RRqgebGaCZQyyr4e?=
 =?us-ascii?Q?fzeGXfpGvxzB7OdNZnLCJhliiWeSWGzwKlhFqfU0vKMu4Irg16y54bH2CUwu?=
 =?us-ascii?Q?2BZX/aDQGuv4gflTro9bVNtfVsFr88bSAmrGkn5E6fo350fB0hFiKZCjP6YV?=
 =?us-ascii?Q?iwCJU1haoZ3iRvcyn+9R5sBwcHUJACr7v77/5ygikv9V8eATG/+100PmdKuy?=
 =?us-ascii?Q?KbDoYVBx22kjNw8A6F+3afWJv5xp3ipYiJwBYVHI4V+qQjvasTrjacWs9AM0?=
 =?us-ascii?Q?BfPhTNUUlaMWbgHIHClOC8RY1UuvHdL7n44TjX2FoEDqS5qvUy1tsgyKIUc9?=
 =?us-ascii?Q?HtOqw/mG1GAjhYBAQod2782mo+poxZNxg1ROZ+UQCl4m7QFlMB3z12KEKwg1?=
 =?us-ascii?Q?Mhab27nMrS4wF9duw3G7RHfBj6A9bITQC0iIz7jEiGkRrRQ+i7qw/3UznFZQ?=
 =?us-ascii?Q?yLMqlaOR1nxWq55oWmsluy7duZ6PM2x9wEGRV8v8HG2x7lPRtZZ1EQD9hPOH?=
 =?us-ascii?Q?nopaDHVGV3doKwdVQGpsgb0JPu2LATZGT8v/gkSEanUzt+n9uzx2AWfjHsGt?=
 =?us-ascii?Q?AOE0RgAah4vej/VHb2thCTiLz02gfYpZty4vSlxEr0gZe5H/6n5lCvbUkqBL?=
 =?us-ascii?Q?+cUEtpwx62C+Vw3+fA7LkVg50Bl6MhVKJSbknsHVSBnYqxOM9SrY95jjWvW3?=
 =?us-ascii?Q?bAxStJvfobrh8GPZa+DaAcoIhsUeOMhRvDjJRtCrhLiZUTDnzjKZm5TNH0QP?=
 =?us-ascii?Q?Cemvc0pIkzykk15Ji+5KJJAeVW+X2LgS9R04nHN3XmhNLNrLIiLzPXUfU9C0?=
 =?us-ascii?Q?1HlC3ASjyHgs492oHnFcH+PUppMoRgzI1BGPTo407KHwRfuHA4jv74gFEXoN?=
 =?us-ascii?Q?P4VT5o3zEL0uzN0G1z6Zem7LCYUK/OvjFtHkZctQyMLC8wRNrtSDgpTybTT0?=
 =?us-ascii?Q?OSd3jIkQKvaGVe++s/dWthiBBT8kjyH5FPvP0QYTbBdEYqLIV+3gLV8l9G9i?=
 =?us-ascii?Q?M75TlR+9Kv0pFH6Mm9IIt8v8KgAV0wELzOyv01CnjZKIJlhCPgqzbxV/Z+4q?=
 =?us-ascii?Q?+Q=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: dknCfbhcfT8m3V/Vo1vwWC+QXXcDBL7JLerSBP6pCjelpqTkUmiWob9PxSyPXI7tHyHzu4Vf692TmyIDpqW/AP9ahjftDceS82xnFPpjehf6lLVZuzYRvBPCbi4qCCfll3ZUn7LWXDKfSjl8LNACh23fs2hsM/X5qqG/mOW5BFmAv9Kw0KTEd0KsAonmQC5OW30989zbPlxfwVjSWe9uLmrvu0Od51lH7pzrA+hW2sE61GByJRdLqlhUVHrTplKRkdLhD/QUR5byqNb7MRTUcMYLF+8dhyuuLGP+9regLZGjk9BxCeCBKWlw0nHLWblSf938S+v4d14xhf/1MakXCTDqh9hchnrfRsvOz7L4sYS5DXuxXhBktxamibk3kCd49yqFlv0s2Zrm9LpSmQqx+BwVBwOrPKB8inB9XTqVGnJ80yxxvvk/rB/s/IB+496NdSkpw7ulBlONx6uWXdumec+dvUP66LPnA+vmBW9rSaJgIIKjIgwrZPwwJnrdrzy15n+SUWF0zeEXzm5WyaFxeecLW/BU0jJ+tJroTtdwtQEXAzOmHSLUBFvWbRHVoK0+VbNg+tMPK7FTaSXacALhF9p+6Du9mLv2O+Lc0UmxRvnK5UCho8GkTq74amoWH3PnskzDmuH8Zj3JBWssQTIVwbxs2dFqk8oWEyxiu2p/vOLtJ41UahR8P7iAqKoUVWBV3NDcx3gQG5K83Uf3S/QrvIM0TS9aUHUmxsLgC9qyCV22kspzXXU5z+AeRv2ewfBZ4fqmGfAchBB3cs8AWV98BQpN3zKZYZa3fdoxD+PbMrzKh59CyMitWC1J+A3zw+ji5f2Yq4UG2wnouatwUfeTaG24hluZQHX+Iudaz5hprkVfGbUHi+rQrJ7HHtDZcGxE
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2aa4c6d6-6820-4957-96a1-08db20079dd8
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Mar 2023 19:02:08.9881
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: JD3FasSCthaUYxwsY9rBjMG47v2NusheX4TAzcTWS8t9rVj07PKOHyZ/Xbj3bhdQjhYvKsmd2gEsQC7H1Hdx4A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR10MB7339
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22
 definitions=2023-03-08_12,2023-03-08_03,2023-02-09_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 adultscore=0 bulkscore=0
 suspectscore=0 mlxlogscore=999 phishscore=0 spamscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000
 definitions=main-2303080162
X-Proofpoint-ORIG-GUID: hBHOSkR56TBsAuMjNtmR-C4otv-qawFk
X-Proofpoint-GUID: hBHOSkR56TBsAuMjNtmR-C4otv-qawFk
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 03/06/23 16:40, Mike Kravetz wrote:
> On 03/06/23 19:15, Peter Xu wrote:
> > On Mon, Mar 06, 2023 at 03:57:30PM -0800, Mike Kravetz wrote:
> > > 
> > > Just wondering if there is anything better or more selective that can be
> > > done?  Does it make sense to have THP backed stacks by default?  If not,
> > > who would be best at disabling?  A couple thoughts:
> > > - The kernel could disable huge pages on stacks.  libpthread/glibc pass
> > >   the unused flag MAP_STACK.  We could key off this and disable huge pages.
> > >   However, I'm sure there is somebody somewhere today that is getting better
> > >   performance because they have huge pages backing their stacks.
> > > - We could push this to glibc/libpthreads and have them use
> > >   MADV_NOHUGEPAGE on thread stacks.  However, this also has the potential
> > >   of regressing performance if somebody somewhere is getting better
> > >   performance due to huge pages.
> > 
> > Yes it seems it's always not safe to change a default behavior to me.
> > 
> > For stack I really can't tell why it must be different here.  I assume the
> > problem is the wasted space and it exaggerates easily with N-threads.  But
> > IIUC it'll be the same as thp to normal memories iiuc, e.g., there can be a
> > per-thread mmap() of 2MB even if only 4K is used each, then if such mmap()
> > is populated by THP for each thread there'll also be a huge waste.

I may be alone in my thinking here, but it seems that stacks by their nature
are not generally good candidates for huge pages.  I am just thinking about
the 'normal' use case where stacks contain local function data and arguments.
Am I missing something, or are huge pages really a benefit here?

Of course, I can imagine some thread with a large amount of frequently
accessed data allocated on it's stack which could benefit from huge
pages.  But, this seems to be an exception rather than the rule.

I understand the argument that THP always means always and everywhere.
It just seems that thread stacks may be 'special enough' to consider
disabling by default.
-- 
Mike Kravetz