Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
Message-ID: <1883d31a-639e-8717-39b1-426628cb0d56@oracle.com>
Date:   Thu, 24 Mar 2022 21:40:05 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.7.0
Subject: Re: scalability regressions related to hugetlb_fault() changes
Content-Language: en-US
To:     Ray Fucillo <Ray.Fucillo@intersystems.com>
Cc:     "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        linux-mm <linux-mm@kvack.org>
References: <D3204B1E-50A1-4261-8C75-3DF77A302502@intersystems.com>
 <43faf292-245b-5db5-cce9-369d8fb6bd21@infradead.org>
 <e4cfa252-7be8-48b2-9f19-019bcc0038af@oracle.com>
 <8E9438A4-56BF-4DBF-9424-2161A488352B@intersystems.com>
From:   Mike Kravetz <mike.kravetz@oracle.com>
In-Reply-To: <8E9438A4-56BF-4DBF-9424-2161A488352B@intersystems.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?clJDaGZaRGprMnhwazMzeHFqUWlLRXRYbTRxcU1SSS91c21KeWNNc2RIS3hR?=
 =?utf-8?B?TGdCYlE2UEF2T3ZzQ0VFRzVzV3NnWDJHRllSTUt2UktVZ2tqS0U1NDVHcnht?=
 =?utf-8?B?VG0vWnlXNitoelZZck9iOFRKNEQ5UitQb1E5cXI3VmZwR0xSeFF2WlZqbDEr?=
 =?utf-8?B?WEN1Ri9jY09DVGg0Mjl0dDg3bW9KMEk2cUZRZ0UzUEQ2U04vSnI0c200R1JE?=
 =?utf-8?B?bHczeVFhay8xcGhrcjVTVnB6YTZqWE0zdjdFRWxTWmx3YjFweldvUVhVRzNS?=
 =?utf-8?B?RXZJMy9lVlBxUkhIZ1crdkI5NGJnRE1ZUUFPWk8yWHZ4NGlRVFVhcUZSS2Rn?=
 =?utf-8?B?R2huK0syZDZ0YlJHS1lQRGxMZk5KbXlHYWRJTGZxVFNMdTRvQ3ZxREZHd0hi?=
 =?utf-8?B?RXA3RlVqMSsrbmpKK1NDM1Npb3phVmxlcXc4NzR4NWJrc1F5QXRaU2dCZFRu?=
 =?utf-8?B?cTVxV09qL0tKZklVd2Q0RmNxOU5wMS9xU2ttcWQvNzgvT0tIV0hoV3VDdXVw?=
 =?utf-8?B?eVFBTDdMRUxpeWVHR2RIUHN2ajFBMEh3bVFBZDEwQVRkc0RMc2pLZDFPYi9C?=
 =?utf-8?B?MHpYL1MrSDI4bVI0TWdtb1R6NjVrc1JGSFlib0VXR0R1ZWsvekJ5VE5tcnFY?=
 =?utf-8?B?VGhIbGNZL1YyYWlrT0lLcHk1aVZNWWY2SUgyTnJJRW9pM3VVZ0NudFloN0g0?=
 =?utf-8?B?TVdVV1pDU0xjMk5FRnJyZ1dpTG9zYjRIcDRZbGNBMmwvdXNFbzlDUnhKaFls?=
 =?utf-8?B?NHB5c2RLTXMxUWdWY2JxT3d4L1NHZzdzWmxBaUlJR2ZmRlZIT2pqeXVXZDQ4?=
 =?utf-8?B?NXR2RDhrTWhYTzA5YmVxL003OEJpWjZXdmdRbE9zQytTMUNGek1BZ090WEdn?=
 =?utf-8?B?WldRdlQ0TGZXeXlTZkRsRHo1N3JsaUVJeWdTcC96M2dGSENWTUZkRUdDa0RM?=
 =?utf-8?B?R0cvSml4TnVBRExEUW8wYXJBSkFzWTRuVTltaWQ4dVAyTE1EZEUrOE9HRUhS?=
 =?utf-8?B?R3p5YTlyc2FZajNMQ2V3UzkrVTVFYktMQmUvTFR5cDlrRjZqUXN6TmtsZjRL?=
 =?utf-8?B?ckFmdzlxUFZSSGJRSitKOVJsRzBKUGpMaDlKSFRkcXZZekNsMDAvKzZCd3Vr?=
 =?utf-8?B?UWZmR2ZnRDNuUU5aUW14VmFsYm5xM1B2RlFlNk52dk1aTHVmVDRwUFRLUUNH?=
 =?utf-8?B?cW5mckZpQ1VCc0hMTEpTajRseTFMMWFGWjVabURQanJsR1k4N09FVU5HaytN?=
 =?utf-8?B?MEN5bU9FVFZOYWdnekZUZlMxOGVvbzNKTGlKRnE4MjhmUitTaTV3U0IyQzFy?=
 =?utf-8?B?dHI4MjVzZnpVcVlpdmt0NjB4OERDWjNNRjFKNEl0V29VYUF1S0ZtaGphWWhP?=
 =?utf-8?B?OXFKaUk3dDc2dnR5SEhSZ1ErZUZpS0czUzRHNHNtaEtxVFlOQWkxWFpISnBR?=
 =?utf-8?B?YmxSUS9KWUUxNUxZQU5HbFIwTEk2UGh3WGNhYmd1WFNjVksrOGE5MjJIdjFL?=
 =?utf-8?B?dmNsOGd0WFdEYWRxYWV3Q05hV0FFcTAzUkR2djN6Q0xKVVlpcmNqdW1KNEUz?=
 =?utf-8?B?NUlHYjNvbjNEdVNwNVJUM1plMkJ6dU8ybVlyTENMdlRieWpCdzFKRXFJdnk3?=
 =?utf-8?B?eG5ySkt3dUtXaTdvajVkYWJrYkdsTldFeHVVVDcwSHIycHZZaVdWRzJLa0Vy?=
 =?utf-8?B?UGlyeFV1V1FuQ250M1hXajRjWDNwQVpPRXUvUnhsRDlzVTZYTyszK01vdGxK?=
 =?utf-8?B?TW5IUWxFMzhodllWUTRKczkyQUlrU1A1STRVWWVjYWcrSFBWMnVZeXJkRjdi?=
 =?utf-8?B?QmhHMHlBbEcrZno0U3RUWm1lU2E4QXovYllLNG9FQ2VocCtNa3J6Z0Y4cldn?=
 =?utf-8?B?VVh5eHBNNE5IMUY0U0Vabi9wYzdydHVKYUdhRTVLcXF6NWlud2pmbFFGTmZZ?=
 =?utf-8?B?aE8vcXF3S004OEtZd25ESVFvMnJyQjZMQllwY1VGVTlqZ0FYK3ZHSGpKSS9z?=
 =?utf-8?B?eUVPSTFqZkErdFcrbE9UVkdMbmw2YlAraWpjcHcxK0dRUnBOc3p2VlNiRjRk?=
 =?utf-8?B?dFdvMW5IUHlzSW84VHlHT2s3Wkg0V1JzbHdQRFpOWVVrcHQyZGlmd3NybW95?=
 =?utf-8?B?WHpIam54VzlBem5nQVlYYTZFb0RMMmRHdmE5ek5GZ3QwdHQ2akRhVzVtZjBL?=
 =?utf-8?B?aURmYW5ScU8rNHY0Z01ub3h5ajJIQ2lBdXkwQWx5dmlwTDk4Vmt1cGF0UVI1?=
 =?utf-8?B?TDNxZ09EUFkwak5PbDI2Q3pwTGNHTVZCcy90UU91cHNkWVBqRkxmTTNyQ3cz?=
 =?utf-8?B?dHFXVVRZaG56Nm5FZ3RVYnFrL3pqajhrTTJKdGdzYll1dEVObzM3S3pEZlpn?=
 =?utf-8?Q?ZgvIl71NVcJigxN8=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 34095d78-9575-48c1-8640-08da0e1989c0
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2022 04:40:07.7707
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: iRtogVcpHdrGcLSork/BYlyCnpPPpm79IBI6EO6XLDDcQno/mqk91fcfvRCjdRhLZrloSDSsoN4aEuwAzpO8mw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB5010
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10296 signatures=694973
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 suspectscore=0 spamscore=0
 mlxlogscore=999 phishscore=0 bulkscore=0 malwarescore=0 mlxscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000
 definitions=main-2203250022
X-Proofpoint-GUID: Jz00IZn-RBALn45jWwAnBuVZvoFvts6n
X-Proofpoint-ORIG-GUID: Jz00IZn-RBALn45jWwAnBuVZvoFvts6n
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,RDNS_NONE,SPF_HELO_NONE,
	T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 3/24/22 17:02, Ray Fucillo wrote:
> 
>> On Mar 24, 2022, at 6:41 PM, Mike Kravetz <mike.kravetz@oracle.com> wrote:
>>
>> I also seem to remember thinking about the possibility of
>> avoiding the synchronization if pmd sharing was not possible.  That may be
>> a relatively easy way to speed things up.  Not sure if pmd sharing comes
>> into play in your customer environments, my guess would be yes (shared
>> mappings ranges more than 1GB in size and aligned to 1GB).
> 
> Hi Mike, 
> 
> This is one very large shared memory segment allocated at database startup.  It's common for it to be hundreds of GB.  We allocate it with shmget() passing SHM_HUGETLB (when huge pages have been reserved for us).  Not sure if that answers...

Yes, so there would be shared pmds for that large shared mapping.  I assume
this is x86 or arm64 which are the only architectures which support shared
pmds.

So, the easy change of "don't take semaphore if pmd sharing is not possible"
would not apply.

>> Also, do you have any specifics about the regressions your customers are
>> seeing?  Specifically what paths are holding i_mmap_rwsem in write mode
>> for long periods of time.  I would expect something related to unmap.
>> Truncation can have long hold times especially if there are may shared
>> mapping.  Always worth checking specifics, but more likely this is a general
>> issue.
> 
> We've seen the write lock originate from calling shmat(), shmdt() and process exit.  We've also seen it from a fork() off of one of the processes that are attached to the shared memory segment.  Some evidence suggests that fork is a more costly case.  However, while there are some important places where we'd use fork(), it's more unusual because most process creation will vfork() and execv() a new database process (which then attaches with shmat()).

Thanks.

I will continue to look at this.  A quick check of the fork code shows the
semaphore held in read mode for the duration of the page table copy.
-- 
Mike Kravetz