Received-SPF: pass (google.com: domain of linux-kernel+bounces-94332-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Precedence: bulk
MIME-Version: 1.0
References: <20240226022452.20558-1-adamli@os.amperecomputing.com>
 <CANn89iLbA4_YdQrF+9Rmv2uVSb1HLhu0qXqCm923FCut1E78FA@mail.gmail.com>
 <a8de785f-8cc3-4075-a5f2-259e20222dcb@os.amperecomputing.com>
 <CANn89iJAKEUu_Fdh0OC-+BJ+iVY0D2y0nAakGLxWZ8TywDu=BA@mail.gmail.com> <11588267-c76d-f0ac-bf98-1875e07b58cb@os.amperecomputing.com>
In-Reply-To: <11588267-c76d-f0ac-bf98-1875e07b58cb@os.amperecomputing.com>
From: Eric Dumazet <edumazet@google.com>
Date: Wed, 6 Mar 2024 18:28:09 +0100
Message-ID: <CANn89i+SiQXeDCKp9yjTz6ReQKxfHn6vHLcWFbCjjW14BiyAbQ@mail.gmail.com>
Subject: Re: [PATCH] net: make SK_MEMORY_PCPU_RESERV tunable
To: "Lameter, Christopher" <cl@os.amperecomputing.com>
Cc: Adam Li <adamli@os.amperecomputing.com>, corbet@lwn.net, davem@davemloft.net, 
	kuba@kernel.org, pabeni@redhat.com, willemb@google.com, 
	yangtiezhu@loongson.cn, atenart@kernel.org, kuniyu@amazon.com, 
	wuyun.abel@bytedance.com, leitao@debian.org, alexander@mihalicyn.com, 
	dhowells@redhat.com, paulmck@kernel.org, joel.granados@gmail.com, 
	urezki@gmail.com, joel@joelfernandes.org, linux-doc@vger.kernel.org, 
	linux-kernel@vger.kernel.org, netdev@vger.kernel.org, 
	patches@amperecomputing.com, shijie@os.amperecomputing.com
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Wed, Mar 6, 2024 at 6:01=E2=80=AFPM Lameter, Christopher
<cl@os.amperecomputing.com> wrote:
>
> On Wed, 28 Feb 2024, Eric Dumazet wrote:
>
> >> __sk_mem_raise_allocated() drops to 0.4%.
> >
> > I suspect some kind of flow/cpu steering issues then.
> > Also maybe SO_RESERVE_MEM would be better for this workload.
>
> This is via loopback. So there is a flow steering issue in the IP
> stack?

Asymmetric allocations / freeing, things that will usually have a high
cost for payload copy anyway.

Maybe a hierarchical tracking would avoid false sharings if some
arches pay a high price to them.

- One per-cpu reserve.    (X MB)

- One per-memory-domain reserve.  (number_of_cpu_in_this_domain * X MB)

- A global reserve, with an uncertainty of number_of_cpus * X MB

Basically reworking lib/percpu_counter.c for better NUMA awareness.