Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32;
Received-SPF: Pass (protection.outlook.com: domain of micron.com designates
 137.201.242.130 as permitted sender) receiver=protection.outlook.com;
 client-ip=137.201.242.130; helo=mail.micron.com; pr=C
From:   Ravi Jonnalagadda <ravis.opensrc@micron.com>
To:     <ying.huang@intel.com>
CC:     <akpm@linux-foundation.org>, <aneesh.kumar@linux.ibm.com>,
        <apopple@nvidia.com>, <dave.hansen@intel.com>,
        <gourry.memverge@gmail.com>, <gregkh@linuxfoundation.org>,
        <gregory.price@memverge.com>, <hannes@cmpxchg.org>,
        <linux-cxl@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        <linux-mm@kvack.org>, <mhocko@suse.com>, <rafael@kernel.org>,
        <shy828301@gmail.com>, <tim.c.chen@intel.com>, <weixugc@google.com>
Subject: Re: [RFC PATCH v3 0/4] Node Weights and Weighted Interleave
Date:   Thu, 2 Nov 2023 15:05:42 +0530
Message-ID: <20231102093542.70-1-ravis.opensrc@micron.com>
In-Reply-To: <87a5rw1wu8.fsf@yhuang6-desk2.ccr.corp.intel.com>
References: <87a5rw1wu8.fsf@yhuang6-desk2.ccr.corp.intel.com>
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Nov 2023 09:35:50.6442
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 1ad04d71-4ffa-4079-5d79-08dbdb871a12
X-MS-Exchange-CrossTenant-Id: f38a5ecd-2813-4862-b11b-ac1d563c806f
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f38a5ecd-2813-4862-b11b-ac1d563c806f;Ip=[137.201.242.130];Helo=[mail.micron.com]
X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000044F3.namprd05.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR08MB6884
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Thu, 02 Nov 2023 02:36:58 -0700 (PDT)

Should Node based interleave solution be considered complex or not would pr=
obably=0D
depend on number of numa nodes that would be present in the system and whet=
her=0D
we are able to setup the default weights correctly to obtain optimum bandwi=
dth=0D
expansion.=0D
=0D
>=0D
>> Pros and Cons of Memory Tier based interleave:=0D
>> Pros:=0D
>> 1. Programming weight per initiator would apply for all the nodes in the=
 tier.=0D
>> 2. Weights can be calculated considering the cumulative bandwidth of all=
=0D
>> the nodes in the tier and need to be programmed once for all the nodes i=
n a=0D
>> given tier.=0D
>> 3. It may be useful in cases where numa nodes with similar latency and b=
andwidth=0D
>> characteristics increase, possibly with pooling use cases.=0D
>=0D
>4. simpler.=0D
>=0D
>> Cons:=0D
>> 1. If nodes with different bandwidth and latency characteristics are pla=
ced=0D
>> in same tier as seen in the current mainline kernel, it will be difficul=
t to=0D
>> apply a correct interleave weight policy.=0D
>> 2. There will be a need for functionality to move nodes between differen=
t tiers=0D
>> or create new tiers to place such nodes for programming correct interlea=
ve weights.=0D
>> We are working on a patch to support it currently.=0D
>=0D
>Thanks!  If we have such system, we will need this.=0D
>=0D
>> 3. For systems where each numa node is having different characteristics,=
=0D
>> a single node might end up existing in different memory tier, which woul=
d be=0D
>> equivalent to node based interleaving.=0D
>=0D
>No.  A node can only exist in one memory tier.=0D
=0D
Sorry for the confusion what i meant was, if each node is having different =
=0D
characteristics, to program the memory tier weights correctly we need to pl=
ace=0D
each node in a separate tier of it's own. So each memory tier will contain=
=0D
only a single node and the solution would resemble node based interleaving.=
=0D
=0D
>=0D
>> On newer systems where all CXL memory from different devices under a=0D
>> port are combined to form single numa node, this scenario might be=0D
>> applicable.=0D
>=0D
>You mean the different memory ranges of a NUMA node may have different=0D
>performance?  I don't think that we can deal with this.=0D
=0D
Example Configuration: On a server that we are using now, four different=0D
CXL cards are combined to form a single NUMA node and two other cards are=0D
exposed as two individual numa nodes.=0D
So if we have the ability to combine multiple CXL memory ranges to a=0D
single NUMA node the number of NUMA nodes in the system would potentially=0D
decrease even if we can't combine the entire range to form a single node.=0D
=0D
>=0D
>> 4. Users may need to keep track of different memory tiers and what nodes=
 are present=0D
>> in each tier for invoking interleave policy.=0D
>=0D
>I don't think this is a con.  With node based solution, you need to know=0D
>your system too.=0D
>=0D
>>>=0D
>>>> Could you elaborate on the 'get what you pay for' usecase you=0D
>>>> mentioned?=0D
>>>=0D
>=0D
>--=0D
>Best Regards,=0D
>Huang, Ying=0D
--=0D
Best Regards,=0D
Ravi Jonnalagadda=