Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
Message-ID: <8516237f-c1a0-aefa-274a-9f8794ae0ccd@linux.ibm.com>
Date:   Wed, 8 Jun 2022 19:50:11 +0530
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.10.0
Subject: Re: [PATCH v5 0/9] mm/demotion: Memory tiers and demotion
Content-Language: en-US
To:     Johannes Weiner <hannes@cmpxchg.org>
Cc:     linux-mm@kvack.org, akpm@linux-foundation.org,
        Wei Xu <weixugc@google.com>, Huang Ying <ying.huang@intel.com>,
        Greg Thelen <gthelen@google.com>,
        Yang Shi <shy828301@gmail.com>,
        Davidlohr Bueso <dave@stgolabs.net>,
        Tim C Chen <tim.c.chen@intel.com>,
        Brice Goglin <brice.goglin@gmail.com>,
        Michal Hocko <mhocko@kernel.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Hesham Almatary <hesham.almatary@huawei.com>,
        Dave Hansen <dave.hansen@intel.com>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Alistair Popple <apopple@nvidia.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Feng Tang <feng.tang@intel.com>,
        Jagdish Gediya <jvgediya@linux.ibm.com>,
        Baolin Wang <baolin.wang@linux.alibaba.com>,
        David Rientjes <rientjes@google.com>
References: <20220603134237.131362-1-aneesh.kumar@linux.ibm.com>
 <YqCq0cUCnQmS6SV4@cmpxchg.org>
From:   Aneesh Kumar K V <aneesh.kumar@linux.ibm.com>
In-Reply-To: <YqCq0cUCnQmS6SV4@cmpxchg.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 6/8/22 7:27 PM, Johannes Weiner wrote:
> Hi Aneesh,
> 
> On Fri, Jun 03, 2022 at 07:12:28PM +0530, Aneesh Kumar K.V wrote:
>> * The current tier initialization code always initializes
>>    each memory-only NUMA node into a lower tier.  But a memory-only
>>    NUMA node may have a high performance memory device (e.g. a DRAM
>>    device attached via CXL.mem or a DRAM-backed memory-only node on
>>    a virtual machine) and should be put into a higher tier.
> 
> I have to disagree with this premise. The CXL.mem bus has different
> latency and bandwidth characteristics. It's also conceivable that
> cheaper and slower DRAM is connected to the CXL bus (think recycling
> DDR4 DIMMS after switching to DDR5). DRAM != DRAM.
> 
> Our experiments with production workloads show regressions between
> 15-30% in serviced requests when you don't distinguish toptier DRAM
> from lower tier DRAM. While it's fixable with manual tuning, your
> patches would bring reintroduce this regression it seems.
> 
> Making tiers explicit is a good idea, but can we keep the current
> default that CPU-less nodes are of a lower tier than ones with CPU?
> I'm having a hard time imagining where this wouldn't be true... Or why
> it shouldn't be those esoteric cases that need the manual tuning.

This was mostly driven by virtual machine configs where we can find 
memory only NUMA nodes depending on the resource availability in the 
hypervisor.

Will these CXL devices be initialized by a driver? For example, if they 
are going to be initialized via dax kmem, we already consider them lower 
memory tier as with this patch series.

-aneesh