Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: Percpu variables, benchmarking, and performance weirdness
To:     Tejun Heo <tj@kernel.org>,
        Jesper Dangaard Brouer <brouer@redhat.com>
Cc:     =?UTF-8?B?QmrDtnJuIFTDtnBlbA==?= <bjorn.topel@gmail.com>,
        bpf <bpf@vger.kernel.org>, LKML <linux-kernel@vger.kernel.org>,
        Christoph Lameter <cl@linux.com>,
        Dennis Zhou <dennis@kernel.org>
References: <CAJ+HfNgNAzvdBw7gBJTCDQsne-HnWm90H50zNvXBSp4izbwFTA@mail.gmail.com>
 <20191220103420.6f9304ab@carbon>
 <20191220151239.GE2914998@devbig004.ftw2.facebook.com>
From:   Eric Dumazet <eric.dumazet@gmail.com>
Message-ID: <a66e79b1-41a8-08f6-8dc2-37ce7a5fff53@gmail.com>
Date:   Fri, 20 Dec 2019 08:22:02 -0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.2.2
MIME-Version: 1.0
In-Reply-To: <20191220151239.GE2914998@devbig004.ftw2.facebook.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


On 12/20/19 7:12 AM, Tejun Heo wrote:
> On Fri, Dec 20, 2019 at 10:34:20AM +0100, Jesper Dangaard Brouer wrote:
>>> So, my question to the uarch/percpu folks out there: Why are percpu
>>> accesses (%gs segment register) more expensive than regular global
>>> variables in this scenario.
>>
>> I'm also VERY interested in knowing the answer to above question!?
>> (Adding LKML to reach more people)
> 
> No idea.  One difference is that percpu accesses are through vmap area
> which is mapped using 4k pages while global variable would be accessed
> through the fault linear mapping.  Maybe you're getting hit by tlb
> pressure?

I definitely seen expensive per-cpu updates in the stack.
(SNMP counters, or per-cpu stats for packets/bytes counters)

It might be nice to have an option to use 2M pages.

(I recall sending some patches in the past about using high-order pages for vmalloc,
but this went nowhere)