Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Thu, 3 Sep 2020 14:18:47 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Matthew Wilcox <willy@infradead.org>
CC:     Roman Gushchin <guro@fb.com>, Michal Hocko <mhocko@suse.com>,
        Zi Yan <ziy@nvidia.com>, <linux-mm@kvack.org>,
        Rik van Riel <riel@surriel.com>,
        "Kirill A . Shutemov" <kirill.shutemov@linux.intel.com>,
        Shakeel Butt <shakeelb@google.com>,
        Yang Shi <yang.shi@linux.alibaba.com>,
        David Nellans <dnellans@nvidia.com>,
        <linux-kernel@vger.kernel.org>
Subject: Re: [RFC PATCH 00/16] 1GB THP support on x86_64
Message-ID: <20200903171847.GJ1152540@nvidia.com>
References: <20200902180628.4052244-1-zi.yan@sent.com>
 <20200903073254.GP4617@dhcp22.suse.cz>
 <20200903162527.GF60440@carbon.dhcp.thefacebook.com>
 <20200903165051.GN24045@ziepe.ca>
 <20200903170157.GE14765@casper.infradead.org>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20200903170157.GE14765@casper.infradead.org>
MIME-Version: 1.0
X-MS-Exchange-MessageSentRepresentingType: 1
Received: from 255.255.255.255 (255.255.255.255) by BL0PR02CA0131.namprd02.prod.outlook.com (2603:10b6:208:35::36) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.16 via Frontend Transport; Thu, 3 Sep 2020 17:18:48 +0000
Received: from jgg by mlx with local (Exim 4.94)        (envelope-from <jgg@nvidia.com>)        id 1kDssp-006dPh-Fy; Thu, 03 Sep 2020 14:18:47 -0300
X-Originating-IP: [156.34.48.30]
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 1086d530-b876-4ec3-2df9-08d8502d6c10
X-MS-TrafficTypeDiagnostic: DM5PR12MB1547:
X-MS-Exchange-Transport-Forked: True
X-Microsoft-Antispam-PRVS: <DM5PR12MB1547BC1807C5BD078196D415C22C0@DM5PR12MB1547.namprd12.prod.outlook.com>
X-MS-Oob-TLC-OOBClassifiers: OLM:10000;
X-MS-Exchange-SenderADCheck: 1
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: hxHMlZ28dUD6yVihd8lADbKNBph4pWofqG3CTH8+cDfEmuz8TE3yH92GKUsUfxP3O+s/HL9BXWRCkky7AUXrBCJMaAk7iC2mOpy0gFzsiY0Fejk7zErtxoomy/NdSjQiq1zgO0NJcgALzax48zDoe7QXA+Y7pzsEf4ojRW3cmCaU4jNuyEvXiHi68YIKqEk4Ex7UNn/SBCMwU744+iEurAs2+sUXVOJRCxg6ak1WVRuIafe2+RIbAY8Rv/dXuHGXscQZowAjO8XA4AsObM1evaSgdVvarFZaruwH3WXyMn6oMzIp5ZL4OcL62hXhS4autjRhpjKYP2n5ZTXyHy9/tcgufppFIFKOPZsK7qQm1JrsmTLZtNr0mCp6iCAo2E731bP461irIefol9Mzdq50e3Ax9gfbqg+FkDKbUDEWuZJFNrLJqM64PEsPcc59mRW1acmkA54CjXLcsfmPIMkmLUE25J4ZXb/TaC48+DT8CYM=
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3834.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(136003)(346002)(366004)(396003)(39860400002)(376002)(426003)(8936002)(36756003)(2616005)(9786002)(316002)(9746002)(26005)(33656002)(2906002)(966005)(186003)(66476007)(66556008)(6916009)(8676002)(1076003)(54906003)(4326008)(83380400001)(86362001)(478600001)(19273905006)(66946007)(5660300002)(562404015)(563064011);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData: 8PpAtcjxjQ+bDuinFvfstHkr/tuLzVv8A95fcOKoll3zWGHNvKznjfeRkp6tSSDmM+aoi0suRulHVcdloL/Ac8uYriz44JTaLj6RkcOg1xDfjBtpph9xQLa4rjkG+WwaAQNL1ypvtkkp//aOO1QoHp3h2Chmh9/z3F771l0T9z4bxIxCF15DuEW03t64IaNnWVMf6JjQFPFvVT8pZUKl3S3zxNdV+Wh050j0+rsiQjd2HsGluwOQVl5gOaKwA4GnmYDSTHL8iBPkI7CUFep45jgq2QGZC6ZJrBO/9wwRSw0ln+qgo5Om+2zOLGahi8pPktH8yN+I+92+40V77PYCOw5WytHgqG0wnmv2kgjuPrWdfK+tx2ZFMjbYTflB1oIASraHV/0pqn84f3qQ6PXhC6gUS4zgdMi9E9wv26uW9BOsUlMNQ+5nhjv4/Oj8vZqEFKYGTx7PIKTjy7GEcifFgXlYx3FMTq2l0OEq+h/3yDZ0LJRMmJIyfSX/sYgHCWTlIC4epF0x9aThZQmlf+7o+v7u8X6uVpmxFX4bXCmAaszPmJ5Zutp6y8ZITjtVcC3KGK/zbld1035RqId9kG0Mi7XpLT/6zGszOtGnxcLH905enluzbiV4emGK/YZjEdtZlhdQZDT7f8QAaj4T1wBNnw==
X-MS-Exchange-CrossTenant-Network-Message-Id: 1086d530-b876-4ec3-2df9-08d8502d6c10
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3834.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Sep 2020 17:18:49.1744
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: NJig0b5a5UKeZqDUxepCyeESXBXxTfNQyIC5a82F5mMdL7nvAz8gKcOREUCwD6NA
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1547
X-OriginatorOrg: Nvidia.com
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1;
        t=1599153536; bh=wjWc+IIcXziz4xdShvvbh73CGUMpP2NZOkg+EZfLRVQ=;
        h=X-PGP-Universal:ARC-Seal:ARC-Message-Signature:
         ARC-Authentication-Results:Authentication-Results:Date:From:To:CC:
         Subject:Message-ID:References:Content-Type:Content-Disposition:
         In-Reply-To:X-ClientProxiedBy:MIME-Version:
         X-MS-Exchange-MessageSentRepresentingType:X-Originating-IP:
         X-MS-PublicTrafficType:X-MS-Office365-Filtering-Correlation-Id:
         X-MS-TrafficTypeDiagnostic:X-MS-Exchange-Transport-Forked:
         X-Microsoft-Antispam-PRVS:X-MS-Oob-TLC-OOBClassifiers:
         X-MS-Exchange-SenderADCheck:X-Microsoft-Antispam:
         X-Microsoft-Antispam-Message-Info:X-Forefront-Antispam-Report:
         X-MS-Exchange-AntiSpam-MessageData:
         X-MS-Exchange-CrossTenant-Network-Message-Id:
         X-MS-Exchange-CrossTenant-AuthSource:
         X-MS-Exchange-CrossTenant-AuthAs:
         X-MS-Exchange-CrossTenant-OriginalArrivalTime:
         X-MS-Exchange-CrossTenant-FromEntityHeader:
         X-MS-Exchange-CrossTenant-Id:X-MS-Exchange-CrossTenant-MailboxType:
         X-MS-Exchange-CrossTenant-UserPrincipalName:
         X-MS-Exchange-Transport-CrossTenantHeadersStamped:X-OriginatorOrg;
        b=et5dIY6aIjKB5vfn9eeG1BrPG0aYJtVuqreTt64bPz1Qm16/9Fd5wOP0NZtcWaQKg
         5pY51ALPHzxbWjMCkPa/l+JlsvGE+lY/5swQ9vPbPkmysuD6Mqz7yl/tapLOYehfbz
         SNE6gt/DRnTBKB79u9v+Lc2gCVcv+JIE72DtfXIy9TmVvzeX2fmb4qj2F+LIKBhScL
         wCnlLGxNFuUBuQG9QN0mvZxCQxW8GXcK9JD3+ZNXrEDBaUS226xX0GXmfawOOVT5AT
         OCQkbJpelRH3+nD04wzXeMswIC5WTFjnUy4W2+2fz2vlEFq77QETCGsjH3s3qLWcod
         /XM4LcNAM0pzA==
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Sep 03, 2020 at 06:01:57PM +0100, Matthew Wilcox wrote:
> On Thu, Sep 03, 2020 at 01:50:51PM -0300, Jason Gunthorpe wrote:
> > At least from a RDMA NIC perspective I've heard from a lot of users
> > that higher order pages at the DMA level is giving big speed ups too.
> > 
> > It is basically the same dynamic as CPU TLB, except missing a 'TLB'
> > cache in a PCI-E device is dramatically more expensive to refill. With
> > 200G and soon 400G networking these misses are a growing problem.
> > 
> > With HPC nodes now pushing 1TB of actual physical RAM and single
> > applications basically using all of it, there is definately some
> > meaningful return - if pages can be reliably available.
> > 
> > At least for HPC where the node returns to an idle state after each
> > job and most of the 1TB memory becomes freed up again, it seems more
> > believable to me that a large cache of 1G pages could be available?
> 
> You may be interested in trying out my current THP patchset:
> 
> http://git.infradead.org/users/willy/pagecache.git
> 
> It doesn't allocate pages larger than PMD size, but it does allocate pages
> *up to* PMD size for the page cache which means that larger pages are
> easier to create as larger pages aren't fragmented all over the system.

Yeah, I saw that, it looks like a great direction.

> If someone wants to opportunistically allocate pages larger than PMD
> size, I've put some preliminary support in for that, but I've never
> tested any of it.  That's not my goal at the moment.
> 
> I'm not clear whether these HPC users primarily use page cache or
> anonymous memory (with O_DIRECT).  Probably a mixture.

There are defiantly HPC systems now that are filesystem-less - they
import data for computation from the network using things like blob
storage or some other kind of non-POSIX userspace based data storage
scheme.

Jason