Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Ray Fucillo <Ray.Fucillo@intersystems.com>
To:     Mike Kravetz <mike.kravetz@oracle.com>
CC:     Ray Fucillo <Ray.Fucillo@intersystems.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        linux-mm <linux-mm@kvack.org>
Subject: Re: scalability regressions related to hugetlb_fault() changes
Thread-Topic: scalability regressions related to hugetlb_fault() changes
Thread-Index: AQHYP7uAlCiDRvHUGUacjbJUOtVkhKzPVz2AgAANE4CAABZqAA==
Date:   Fri, 25 Mar 2022 00:02:08 +0000
Message-ID: <8E9438A4-56BF-4DBF-9424-2161A488352B@intersystems.com>
References: <D3204B1E-50A1-4261-8C75-3DF77A302502@intersystems.com>
 <43faf292-245b-5db5-cce9-369d8fb6bd21@infradead.org>
 <e4cfa252-7be8-48b2-9f19-019bcc0038af@oracle.com>
In-Reply-To: <e4cfa252-7be8-48b2-9f19-019bcc0038af@oracle.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-ID: <9BD5E1F3232CF04A8B389603D457D492@exchangemail.iscinternal.com>
Content-Transfer-Encoding: 8BIT
MIME-Version: 1.0
Precedence: bulk


> On Mar 24, 2022, at 6:41 PM, Mike Kravetz <mike.kravetz@oracle.com> wrote:
> 
> I also seem to remember thinking about the possibility of
> avoiding the synchronization if pmd sharing was not possible.  That may be
> a relatively easy way to speed things up.  Not sure if pmd sharing comes
> into play in your customer environments, my guess would be yes (shared
> mappings ranges more than 1GB in size and aligned to 1GB).

Hi Mike, 

This is one very large shared memory segment allocated at database startup.  It's common for it to be hundreds of GB.  We allocate it with shmget() passing SHM_HUGETLB (when huge pages have been reserved for us).  Not sure if that answers...

> Also, do you have any specifics about the regressions your customers are
> seeing?  Specifically what paths are holding i_mmap_rwsem in write mode
> for long periods of time.  I would expect something related to unmap.
> Truncation can have long hold times especially if there are may shared
> mapping.  Always worth checking specifics, but more likely this is a general
> issue.

We've seen the write lock originate from calling shmat(), shmdt() and process exit.  We've also seen it from a fork() off of one of the processes that are attached to the shared memory segment.  Some evidence suggests that fork is a more costly case.  However, while there are some important places where we'd use fork(), it's more unusual because most process creation will vfork() and execv() a new database process (which then attaches with shmat()).