Received: by 2002:a25:d7c1:0:0:0:0:0 with SMTP id o184csp1715416ybg; Sat, 19 Oct 2019 00:57:38 -0700 (PDT) X-Google-Smtp-Source: APXvYqxz3+qwwcUB5w31m5MbTGRzWft+WSvXlfILOYOGhPKG78P2iZqr+/0xhuJlYSyC0PBptIBk X-Received: by 2002:a50:ce06:: with SMTP id y6mr14014058edi.259.1571471858711; Sat, 19 Oct 2019 00:57:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1571471858; cv=none; d=google.com; s=arc-20160816; b=wtvVQbjpLeHHlR1mKK4tYVQhZ5CFBu0j6ftKGc+8al0D8tx86WUGotRptbJ9p5tWkB dr1KGadtaZ+RAsbJg7Hetf701fbBPr9eUm2uXlo/7iOg/Pb1+3YseOaZXESc4lPwiiY7 hkxQKpkZLKA1qVKoRvQvhNPm8PkN5r6Pkfq32BqDxApSAj+lYRvwB9ZhIB8xg/90erry omoxMHnjLAVvhqZUKzYkVumJL7oHOTL+kcLKlR0NweFlq+pJMG5LJ9U5m89CDk5TbrBe DdGKe+O1PJjM+mo+nWFyz5WQSi1iXW9Me2gJljBvjvBNd0N/AKXE0f/d9E+5QWXLn9xg Iabw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:dkim-signature; bh=gbrbICubD3bXmWnYbOIskkXx4MwqbmP9fw85/ZSp3Ck=; b=kK8FToUkGpCx3ACdhKRBjWWD9sE5FadiXME7JG0j3nyg9KP4K1c/7yx3I8iBSFdB4F m/v2o5C8pgf0oY6Ao5dOBGQcKAiWlW3i4/XlCh7ndCnjibvowgKHl7W/nJ1az1Ifxp7b 08B8Ivw6sqLXZsrGy6p0z25zzKw89OKZ+PSaenFrVf9XdvV2swg8nrG1VuwyFB7yEL24 BqPW4Ld/qvIbNUk9HcTrLg313ab18icxw/EDmsjhUn4QzoSHHFSyyrs+nQRkmerRypAl 7uAmaY1rzN0WSAZIa0DLUJC708vl4ebjjYebVHddhVsK5qDwG5nyB2JdnsMGphcG/uqv U3Xw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=aIOtObID; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w6si5252478eda.336.2019.10.19.00.57.15; Sat, 19 Oct 2019 00:57:38 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=aIOtObID; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2391860AbfJRILv (ORCPT + 99 others); Fri, 18 Oct 2019 04:11:51 -0400 Received: from mail-yb1-f195.google.com ([209.85.219.195]:36882 "EHLO mail-yb1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727918AbfJRILv (ORCPT ); Fri, 18 Oct 2019 04:11:51 -0400 Received: by mail-yb1-f195.google.com with SMTP id z125so1569955ybc.4 for ; Fri, 18 Oct 2019 01:11:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=gbrbICubD3bXmWnYbOIskkXx4MwqbmP9fw85/ZSp3Ck=; b=aIOtObIDPdGwv7gk6cVYXSD+b7Yu7fjEezPclytEOVEnidkTqz45CNM/rjgEbn3n2Z PlFokea7ndfCUjN77HVdAXcDxcD5PlVPDILTeWKn2L65QHvKY5dBrl1/38VUQNn133y5 J136qxDJcwpF4RkBMisALmKLHRm5SeXnUgD63kM/HWijSQXrt04h9xFBRW8hITaU5CaN NbORJ5V4HRCKYIDet8v49ZOu+zQPIwFYKFMC9srlJ0zQzdA3xLtzqgw9GgKkC7VXiOah elBa5z0y2Rljjdvm5511XrHapDcWom9TATNBKypSVXFSh+OIzuYlcZBQHBSmeOIcL+Hd KLyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=gbrbICubD3bXmWnYbOIskkXx4MwqbmP9fw85/ZSp3Ck=; b=g5pitVomjm+GIou/xQAPLbJ17UIq1+mq2o+X7fHh2H9kXLhBevL+4RW6aY/B+FKhD6 38PUha2uwJnenl7u3e7PUi9VjWE5lsUJSRsCwVIe6/pfwqqijTXT4zuTYCdQ37BBWvP4 /76mla1f1PCXB8ePF7h3vP50gDp7QbgrYMCt64w0NxrINMBpupIAk/k1vlqZVmYd8w4m VWmC98hEAvTYMgegxXZfggoVIyMTsF+8XM6/NuuaEl8U1PeQ7kTKMNz9wEKQTww8deie 7kr5vrpk4vsV8Oqxy8Y9YvpVVRhcLueRqXJdTBnwa2fjTQh2OG8v+llenVnW/5gOV8SH SYqw== X-Gm-Message-State: APjAAAUdUNjWzf0ixO1J+r681zMnBI7z+Xq3/SjZekpDtd4NioKDgp+K dhG8DtpLafqHZmvjEOakzbPnPQBFXR8Oc+ukd0q7VQ== X-Received: by 2002:a25:e5c2:: with SMTP id c185mr5195789ybh.332.1571386309974; Fri, 18 Oct 2019 01:11:49 -0700 (PDT) MIME-Version: 1.0 References: <20191016221148.F9CCD155@viggo.jf.intel.com> <85512332-d9d4-6a72-0b42-a8523abc1b5f@intel.com> In-Reply-To: <85512332-d9d4-6a72-0b42-a8523abc1b5f@intel.com> From: Suleiman Souhlal Date: Fri, 18 Oct 2019 17:11:38 +0900 Message-ID: Subject: Re: [PATCH 0/4] [RFC] Migrate Pages in lieu of discard To: Dave Hansen Cc: Dave Hansen , Linux Kernel , linux-mm@kvack.org, dan.j.williams@intel.com, Shakeel Butt , Jonathan Adams Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Oct 18, 2019 at 1:32 AM Dave Hansen wrote: > > On 10/17/19 9:01 AM, Suleiman Souhlal wrote: > > One problem that came up is that if you get into direct reclaim, > > because persistent memory can have pretty low write throughput, you > > can end up stalling users for a pretty long time while migrating > > pages. > > Basically, you're saying that memory load spikes turn into latency spikes? Yes, exactly. > FWIW, we have been benchmarking this sucker with benchmarks that claim > to care about latency. In general, compared to DRAM, we do see worse > latency, but nothing catastrophic yet. I'd be interested if you have > any workloads that act as reasonable proxies for your latency requirements. Sorry, I don't know of any specific workloads I can share. :-( Maybe Jonathan or Shakeel have something more. I realize it's not very useful without giving specific examples, but even disregarding persistent memory, we've had latency issues with direct reclaim when using zswap. It's been such a problem that we're conducting experiments with not doing zswap compression in direct reclaim (but still doing it proactively). The low write throughput of persistent memory would make this worse. I think the case where we're most likely to run into this is when the machine is close to OOM situation and we end up thrashing rather than OOM killing. Somewhat related, I noticed that this patch series ratelimits migrations from persistent memory to DRAM, but it might also make sense to ratelimit migrations from DRAM to persistent memory. If all the write bandwidth is taken by migrations, there might not be any more available for applications accessing pages in persistent memory, resulting in higher latency. Another issue we ran into, that I think might also apply to this patch series, is that because kernel memory can't be allocated on persistent memory, it's possible for all of DRAM to get filled by user memory and have kernel allocations fail even though there is still a lot of free persistent memory. This is easy to trigger, just start an application that is bigger than DRAM. To mitigate that, we introduced a new watermark for DRAM zones above which user memory can't be allocated, to leave some space for kernel allocations. -- Suleiman