Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Thu, 3 Sep 2020 13:50:51 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Roman Gushchin <guro@fb.com>
CC:     Michal Hocko <mhocko@suse.com>, Zi Yan <ziy@nvidia.com>,
        <linux-mm@kvack.org>, Rik van Riel <riel@surriel.com>,
        "Kirill A . Shutemov" <kirill.shutemov@linux.intel.com>,
        Matthew Wilcox <willy@infradead.org>,
        Shakeel Butt <shakeelb@google.com>,
        Yang Shi <yang.shi@linux.alibaba.com>,
        David Nellans <dnellans@nvidia.com>,
        <linux-kernel@vger.kernel.org>
Subject: Re: [RFC PATCH 00/16] 1GB THP support on x86_64
Message-ID: <20200903165051.GN24045@ziepe.ca>
References: <20200902180628.4052244-1-zi.yan@sent.com>
 <20200903073254.GP4617@dhcp22.suse.cz>
 <20200903162527.GF60440@carbon.dhcp.thefacebook.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20200903162527.GF60440@carbon.dhcp.thefacebook.com>
MIME-Version: 1.0
X-MS-Exchange-MessageSentRepresentingType: 1
Received: from 255.255.255.255 (255.255.255.255) by BL0PR0102CA0007.prod.exchangelabs.com (2603:10b6:207:18::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.15 via Frontend Transport; Thu, 3 Sep 2020 16:50:53 +0000
Received: from jgg by mlx with local (Exim 4.94)        (envelope-from <jgg@nvidia.com>)        id 1kDsRn-006cqK-F0; Thu, 03 Sep 2020 13:50:51 -0300
X-Originating-IP: [156.34.48.30]
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: ae91dc40-94a1-453f-4e7d-08d850298563
X-MS-TrafficTypeDiagnostic: DM6PR12MB4483:
X-MS-Exchange-Transport-Forked: True
X-Microsoft-Antispam-PRVS: <DM6PR12MB4483FE1B68A4DB97CB9E182DC22C0@DM6PR12MB4483.namprd12.prod.outlook.com>
X-MS-Oob-TLC-OOBClassifiers: OLM:7691;
X-MS-Exchange-SenderADCheck: 1
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: Eic8y0lNbklrNx7MzUawSUDvvjlE+fViJbbPWbtaO/FXRP4B3bRZyKD7V3yY/u0cUNlbQhrsLht3sYFHaPrMIanr2/A031O+zYIeDRpTyRsXkP62f6ZzznOpLvbY4wim88o+POM8wGeMDJyFeTg4rq9heNJZey464kshNUtxypPyT0rpdnLH86s1Bu0xf9ZBzrgZbkRz8y1OSyRkyrojoaHaBE8XczQcB9KTikLnqSaivpHi4pbaBUHY073jQzoMKtKonGyv8gIY9AXpmBwlzDhhBJLiime5FwBo/3cdePIRo8MoZNd2M+1951klqnMeaqUY1t01pY2E0GDmOiL/Dg==
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3834.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(396003)(136003)(366004)(39860400002)(376002)(346002)(8676002)(66476007)(9686003)(5660300002)(478600001)(9746002)(86362001)(83380400001)(6916009)(8936002)(36756003)(9786002)(316002)(54906003)(426003)(66556008)(26005)(1076003)(2906002)(186003)(4326008)(33656002)(66946007);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData: q25V+VviRFs2Sq3EAnOfv4cWq9WFfp9ZQ71IMI22hX8izeF5cNlFGwl+tPOslTC7aXoi+9y394VLodoZ2LCvIzTPK/shwcQ4wCNFBFnB/W7/YcB85LynNK4QnJwFkGeCjPcTU5mYrSAzg2Vm535cUiVZOOSzDDbShFtUE5Gl6VoVifiihH4OovZgiXGmnQKq6eMU36rQQC/eTMJPaT28825hB6IVW2PO6DzgsnVebdRt6g2pbU1yTu4shtxNmLt3SgrgguHUQ6SiJ8UFZXmYBEJ5et41n0Auxu7LZilCqhxqk0913UcvVekXLmfNZJG1XKbP+gYD9WbPr9t7V6kzME9wIwbEzPIKqG4mnAEM93LZVo1B3VKT4/x1BRJhcVEZnl9tioQkEMKjDrEeMhXgFHMSxlivzHc6DYY5KoebmtMZGfPKDoh6gvYu+CID9vwUe0ulS/3JI6K5UGmkMUR/e/txWM4ILka8nnvrMgRuAxqtB5jMxaBaIzHoK/wuzMckwdS5dV46/Zi3yvRHM4AegV49EDIsBWgU0xtJXKI5PMnMQ6ejlv+ZCubdyTZ59TVEH1gTQL/3UQXT+DvLj/Rv7QhnNdFz2ZkNrGgZNZ2lyPFk39mGR2x0QshSoodxGk61aEbliLGm2H3lg3vSPtAouQ==
X-MS-Exchange-CrossTenant-Network-Message-Id: ae91dc40-94a1-453f-4e7d-08d850298563
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3834.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Sep 2020 16:50:53.6763
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: IbIcmPgetJVZSKy9790CuL0FhpUi+lQu447zYJG3+DtkXm9RSoHXjECy5cafyrZw
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4483
X-OriginatorOrg: Nvidia.com
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1;
        t=1599151858; bh=VAqB/F7YFynuW/Opn3oKNGc751GDqu4WIOi0AQX8QDQ=;
        h=X-PGP-Universal:ARC-Seal:ARC-Message-Signature:
         ARC-Authentication-Results:Authentication-Results:Date:From:To:CC:
         Subject:Message-ID:References:Content-Type:Content-Disposition:
         In-Reply-To:X-ClientProxiedBy:MIME-Version:
         X-MS-Exchange-MessageSentRepresentingType:X-Originating-IP:
         X-MS-PublicTrafficType:X-MS-Office365-Filtering-Correlation-Id:
         X-MS-TrafficTypeDiagnostic:X-MS-Exchange-Transport-Forked:
         X-Microsoft-Antispam-PRVS:X-MS-Oob-TLC-OOBClassifiers:
         X-MS-Exchange-SenderADCheck:X-Microsoft-Antispam:
         X-Microsoft-Antispam-Message-Info:X-Forefront-Antispam-Report:
         X-MS-Exchange-AntiSpam-MessageData:
         X-MS-Exchange-CrossTenant-Network-Message-Id:
         X-MS-Exchange-CrossTenant-AuthSource:
         X-MS-Exchange-CrossTenant-AuthAs:
         X-MS-Exchange-CrossTenant-OriginalArrivalTime:
         X-MS-Exchange-CrossTenant-FromEntityHeader:
         X-MS-Exchange-CrossTenant-Id:X-MS-Exchange-CrossTenant-MailboxType:
         X-MS-Exchange-CrossTenant-UserPrincipalName:
         X-MS-Exchange-Transport-CrossTenantHeadersStamped:X-OriginatorOrg;
        b=EG9FdAIsQQhHGQGqm0EsHLa4rQAYDH1VJuMdHNXk4RI4ihNmNrtr3hhFbLM1hYINm
         XWSD+QmSky1NMEyHXL1heeRBZHGeWuPt5ZIHTSSMdfAOpam2aRfpsiNX50myAQleVf
         s/EVvp7BWdVCaEgC3T9KgmOsONGLU8X4//qAXinrNaDQVxD/dgvXkMUpgBz3F7koRV
         u9LFkPkagJcI7X+32oVn8kA3A7rmt1eRgm6CtKGoZoC8dIGcNHUI0BC7KeMQIft2UF
         3HriNCySylPu/8erDCHjd2BoFnuswKP1pIDRne2g4hStjzja7YymxJ1owsvB4O8MtT
         c7HvO12/zWTdw==
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Sep 03, 2020 at 09:25:27AM -0700, Roman Gushchin wrote:
> On Thu, Sep 03, 2020 at 09:32:54AM +0200, Michal Hocko wrote:
> > On Wed 02-09-20 14:06:12, Zi Yan wrote:
> > > From: Zi Yan <ziy@nvidia.com>
> > > 
> > > Hi all,
> > > 
> > > This patchset adds support for 1GB THP on x86_64. It is on top of
> > > v5.9-rc2-mmots-2020-08-25-21-13.
> > > 
> > > 1GB THP is more flexible for reducing translation overhead and increasing the
> > > performance of applications with large memory footprint without application
> > > changes compared to hugetlb.
> > 
> > Please be more specific about usecases. This better have some strong
> > ones because THP code is complex enough already to add on top solely
> > based on a generic TLB pressure easing.
> 
> Hello, Michal!
> 
> We at Facebook are using 1 GB hugetlbfs pages and are getting noticeable
> performance wins on some workloads.

At least from a RDMA NIC perspective I've heard from a lot of users
that higher order pages at the DMA level is giving big speed ups too.

It is basically the same dynamic as CPU TLB, except missing a 'TLB'
cache in a PCI-E device is dramatically more expensive to refill. With
200G and soon 400G networking these misses are a growing problem.

With HPC nodes now pushing 1TB of actual physical RAM and single
applications basically using all of it, there is definately some
meaningful return - if pages can be reliably available.

At least for HPC where the node returns to an idle state after each
job and most of the 1TB memory becomes freed up again, it seems more
believable to me that a large cache of 1G pages could be available?

Even triggering some kind of cleaner between jobs to defragment could
be a reasonable approach..

Jason