Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: [PATCH] percpu/module resevation: change resevation size iff
 X86_VSMP is set
To:     Christopher Lameter <cl@linux.com>
Cc:     Dennis Zhou <dennis@kernel.org>, Eial Czerwacki <eial@scalemp.com>,
        tj@kernel.org, linux-kernel@vger.kernel.org,
        Shai Fultheim <shai@scalemp.com>,
        Oren Twaig <oren@scalemp.com>,
        "Paul E. McKenney" <paulmck@linux.ibm.com>
References: <1548071251-1849-1-git-send-email-eial@scalemp.com>
 <a7eff2ed-0e80-edae-2a97-a3fdab2d05f5@google.com>
 <20190301203455.GA97188@dennisz-mbp.dhcp.thefacebook.com>
 <c72402f2-967e-cd56-99d8-9139c9e7f267@google.com>
 <010001693b404440-248fa987-624c-4587-940b-56e2ed4226d9-000000@email.amazonses.com>
From:   Barret Rhoden <brho@google.com>
Message-ID: <85726648-82f3-6b6b-a749-03c4159e78f3@google.com>
Date:   Wed, 13 Mar 2019 15:40:04 -0400
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
 Thunderbird/52.9.1
MIME-Version: 1.0
In-Reply-To: <010001693b404440-248fa987-624c-4587-940b-56e2ed4226d9-000000@email.amazonses.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

Hi -

On 03/01/2019 04:54 PM, Christopher Lameter wrote:
> On Fri, 1 Mar 2019, Barret Rhoden wrote:
> 
>> I'm not familiar with VSMP - how bad is it to use L1 cache alignment instead
>> of 4K page alignment?  Maybe some structures can use the smaller alignment?
>> Or maybe have VSMP require SRCU-using modules to be built-in?
> 
> It is very expensive. VMSP exchanges 4K segments via RDMA between servers
> to build a large address space and run a kernel in the large address
> space. Using smaller segments can cause a lot of
> "cacheline" bouncing (meaning transfers of 4K segments back and forth
> between servers).
> 

Given that these are large machines, would it be OK to statically 
reserve 64K on them for modules' percpu data?

The bug that led me to here was from someone running on a non-VSMP 
machine but had that config set.  Perhaps we make it more clear in the 
Kconfig option to not set it on other machines.  That might make it less 
likely anyone on a non-VSMP machine pays the 64K overhead.

Are there any other alternatives?  Not using static SRCU in any code 
that could be built as a module seems a little harsh.

Thanks,

Barret