Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37;
Content-Type: text/plain;
        charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.200.91.1.1\))
Subject: Re: [PATCH v6 0/4] riscv: tlb flush improvements
From:   Nadav Amit <nadav.amit@gmail.com>
In-Reply-To: <mhng-4e3e3fa7-5e25-494c-a3ad-6ef7ec78cf20@palmer-ri-x1c9a>
Date:   Tue, 7 Nov 2023 10:38:01 +0200
Cc:     Alexandre Ghiti <alexghiti@rivosinc.com>,
        Will Deacon <will@kernel.org>,
        "Aneesh Kumar K . V" <aneesh.kumar@linux.ibm.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Nick Piggin <npiggin@gmail.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Mayuresh Chitale <mchitale@ventanamicro.com>,
        Vincent Chen <vincent.chen@sifive.com>,
        Paul Walmsley <paul.walmsley@sifive.com>,
        Albert Ou <aou@eecs.berkeley.edu>, linux-arch@vger.kernel.org,
        linux-mm <linux-mm@kvack.org>, linux-riscv@lists.infradead.org,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Samuel Holland <samuel@sholland.org>,
        Lad Prabhakar <prabhakar.csengg@gmail.com>
Content-Transfer-Encoding: quoted-printable
Message-Id: <D1DBDDD4-9B6A-40EA-993E-FF6CF02A2668@gmail.com>
References: <mhng-4e3e3fa7-5e25-494c-a3ad-6ef7ec78cf20@palmer-ri-x1c9a>
To:     Palmer Dabbelt <palmer@dabbelt.com>
Precedence: bulk


> On Nov 7, 2023, at 9:00=E2=80=AFAM, Palmer Dabbelt =
<palmer@dabbelt.com> wrote:
>=20
> On Mon, 30 Oct 2023 07:01:48 PDT (-0700), nadav.amit@gmail.com wrote:
>>=20
>>> On Oct 30, 2023, at 3:30 PM, Alexandre Ghiti =
<alexghiti@rivosinc.com> wrote:
>>> + on_each_cpu_mask(cmask,
>>> + __ipi_flush_tlb_range_asid,
>>> + &ftd, 1);
>>=20
>> Unrelated, but having fed
>=20
> Do you mean `ftd`?
>=20
> If so I'm not all that convinced that's a problem: sure it's 4x`long`, =
so we pass it on the stack instead of registers, but otherwise we'd need =
another `on_each_cpu_mask()` callback to shim stuff through via =
registers.

I have no idea why you need to move stuff through the registers.

>> Actually, it is best not to put it on the stack, if possible to =
reduce
>> cache traffic.
>=20
> Sorry if I'm just missing something, but I'm not convinced this is a =
measurable performance problem.

I am not going to try to convince you (I ran the numbers on x86 a long
time ago).

There is a cost of bouncing cache-lines (because multiple cores access
the stack), TLB-miss on remote cores (which is mostly avoidable if ftd
is global).

Having said that, the optimizations you added now and intend to add in
the next steps are definitely more important for performance.