Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <bca034e9-5218-5ae4-79df-8c40e0aa6d3d@oracle.com>
Date:   Thu, 14 Jul 2022 16:02:46 -0600
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.9.1
Subject: Re: [PATCH v2 0/9] Add support for shared PTEs across processes
Content-Language: en-US
To:     David Hildenbrand <david@redhat.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Mike Kravetz <mike.kravetz@oracle.com>
Cc:     willy@infradead.org, aneesh.kumar@linux.ibm.com, arnd@arndb.de,
        21cnbao@gmail.com, corbet@lwn.net, dave.hansen@linux.intel.com,
        ebiederm@xmission.com, hagen@jauu.net, jack@suse.cz,
        keescook@chromium.org, kirill@shutemov.name, kucharsk@gmail.com,
        linkinjeon@kernel.org, linux-fsdevel@vger.kernel.org,
        linux-kernel@vger.kernel.org, linux-mm@kvack.org,
        longpeng2@huawei.com, luto@kernel.org, markhemm@googlemail.com,
        pcc@google.com, rppt@kernel.org, sieberf@amazon.com,
        sjpark@amazon.de, surenb@google.com, tst@schoebel-theuer.de,
        yzaikin@google.com
References: <cover.1656531090.git.khalid.aziz@oracle.com>
 <20220701212403.77ab8139b6e1aca87fae119e@linux-foundation.org>
 <0864a811-53c8-a87b-a32d-d6f4c7945caa@redhat.com>
 <357da99d-d096-a790-31d7-ee477e37c705@oracle.com>
 <397f3cb2-1351-afcf-cd87-e8f9fb482059@redhat.com>
From:   Khalid Aziz <khalid.aziz@oracle.com>
In-Reply-To: <397f3cb2-1351-afcf-cd87-e8f9fb482059@redhat.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cjFXdmVxVkt1VXZhN0EwODg2LzVCcDhrMDc5QjY1d2RweEpWMndxMnU0dHpF?=
 =?utf-8?B?ZFF4RW9UUi82b3JoYng4Q1NOcjAvdWcwMHo2ZTdzTGI3VlY3Y2tLTkdEQnBI?=
 =?utf-8?B?TkRBcG53SlAwR0RVSys1eSt0dUNWS0dlai82Vm9QdHRDT0dObnZmb0dFUmsy?=
 =?utf-8?B?VXhSVWRZbzVjUGl5bWxWTVR1a2VibllMUFB5ZzFRUWRUOWZoNWxqOEVyWnNa?=
 =?utf-8?B?M3ZFNVAzZUpjcmI5V2ZQV0NsNDUvS1ZMajhpODZCVnpCVEVQb1NHQTFab0dj?=
 =?utf-8?B?RWZiaXZ4dC80MTFKZXhyVDJwS2NzSHNEMWZoeE40R05oS3h6N05hL2JndFgy?=
 =?utf-8?B?T0gxcmZPNGF6NENPUnUwd3NpSnpyamVhOGY1MThUYm5PSzQyYlRqaWp5UFZz?=
 =?utf-8?B?Z05jR1lvSmo5QmJ5b1ZkVFJzeFRPckhESERpdU5vT1I1NEVzZ0dVRjNkcW54?=
 =?utf-8?B?M01hcHFIa1EzU1U5djg0RUlMUU5ERmZDRU4ybVUvMU0vVHdlSHdDSHJZYkdX?=
 =?utf-8?B?aFpLQ3N4a3h1aThDTExIend4cGtnRlJFSC9qdTJvbXRVZkFhOThnVmtQeGQ2?=
 =?utf-8?B?TnJnNmcxcTVXU1RGL2xhT1I4M09tOTFjU0ZTRzFqTVJmU0lIUnp4MHpqYko3?=
 =?utf-8?B?MEhydVZIdFl1aEwrMzlHeVloSTI0dW0vcHk1WVl5RS9wS3dpVTE5eVRmd2Uw?=
 =?utf-8?B?ZmVwSXlUeGVwdDZkTXQydmV5UzNtRHhtWGprN0xvU1hvREMvelkyVzlBQ3NZ?=
 =?utf-8?B?ZHl3dm9VQzdRbGxzVEhwbWdvSFg4ZVNJMHRwTTJvQ2hEa1lRNmxCZURZL21V?=
 =?utf-8?B?UkM3bFUrMnhpWFhnbXVLbG9VaWcxdTVrUHdYajc3Q0hPZjZ0VUgwSkI5d3M3?=
 =?utf-8?B?M3J0VFgxQ0V4SzczUmpueUdGakhHajB4VVNSdnRsODhDRk5lTjUvQ1hKd3NY?=
 =?utf-8?B?Zm1uakJLSnc0UTlOYjlVS3Z5Vm9adjBXQTZ6VWF2RS9Vc1JETU1CMHg5dWEw?=
 =?utf-8?B?TGkvd1YvRThFdXhmODRiK3R4WVlIWUdlQ1I4WVJ5V1JjQmlscGFKYzBWQmx0?=
 =?utf-8?B?V0V5bHROL09HTytHQ2FCSlFjeTNzbXhqOXhIWElYcmRsOTVtSXFuYzF5VGVs?=
 =?utf-8?B?ZHA5czkzcHpiMDVjczB3QS9PT0R0OHF4MmdhRjBmejQwRENRRWFseVBzUkdI?=
 =?utf-8?B?b1EvcE1qOGloMURZVFN5djJ0cEo0cWQzSUlnNE52R3k0QlhoV2U2dnNmQjRW?=
 =?utf-8?B?aS9OcVJrcnpXRitOTVpVYjJLM0E3cGJZcC9ybkQwSUtFaEp2UmQxWm1TOEZ0?=
 =?utf-8?B?dlRHbkVpOEIzUGticW94TWI5K0E1eXdLVEVBZXFmOGMzVEV0cXMxeGY3VkYy?=
 =?utf-8?B?WHUwMXNrRCt3eTJEVTFZSzNmMGRHMzUwVExrNU1VdHVSNzNJcmhsa05RUFNZ?=
 =?utf-8?B?T2JrckhsUTg3dGJWU3U1YVVuckVwWE1LM1gxVEhJMWsrYmlvaW1jZkZIcXdQ?=
 =?utf-8?B?d2dGb2NLcURsMC8wYy9HME1aYUZVcUxGK0tUSmZPN0dxd2h5UXptd0xyRUw3?=
 =?utf-8?B?QjlUTXpZVUNSbXJiclJ2NFBhdmFOcGk3ZENkdVQwcVRrOEpabVkyNDhLNVBo?=
 =?utf-8?B?VjFGZ2NKV0FKOGtKT3F2UGNMM2ZDSmNZWEM0NFBvcWtSYUJXL1RXS1owaXN2?=
 =?utf-8?B?M0VOckphWW84dWVxc1dRc1B0YlhIYlhGMHExbUxHTVJiUDRXK2VMa215MXRY?=
 =?utf-8?B?d3dFcFVUUDdkZFk3ME1YZU1MY3lIWUFvS3gyYlprNTlnSmhGeTAxQk5lS2NW?=
 =?utf-8?B?Sk5yR0lSdStuTHBSSzdPZnJJM1RVaU96UDVVSUFLWU1CUEhBYUdNajJBSzNE?=
 =?utf-8?B?dHVHMGpVOEo3bXRFOWVXMVc0RTNHblJrcHdYUUVGY2FnUHRoODFNWnZzdEJG?=
 =?utf-8?B?ZFUzb1E0SVdpR0p4c0U4RmFPQStYOTVVZjFyWmwxZG1xVERDWVlRbFdtMEhl?=
 =?utf-8?B?aFVkOXMzNTc5Rk1aMTVHL0NBNkJlWXVlWGJDSnBSaW5QckkyeVp1aXRKVHJU?=
 =?utf-8?B?NGJ5Y0ZvOXplWGxSelhQNXdTL3Bjc0d6U09QMkpnTFJwUjhDei9qdlZQK2hD?=
 =?utf-8?Q?0ZHgUm/aax66mTYsiav41GkIu?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 3a0256a6-d883-4fec-fef0-08da65e498a7
X-MS-Exchange-CrossTenant-AuthSource: BN8PR10MB3220.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jul 2022 22:02:51.7254
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: SHLHl+g5nPYh9hVreQ09rqVg5iqxDYUMxoUrdf06C0wSQwI52hCz4t1b4AMpCUfvfjjYo248NFyD5yggh41PWA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB6103
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.883
 definitions=2022-07-14_17:2022-07-14,2022-07-14 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 spamscore=0
 adultscore=0 suspectscore=0 bulkscore=0 mlxlogscore=999 mlxscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2206140000
 definitions=main-2207140097
X-Proofpoint-ORIG-GUID: KVpRYsG9v7RyAE-d0ix2pOGTlC4QQhxH
X-Proofpoint-GUID: KVpRYsG9v7RyAE-d0ix2pOGTlC4QQhxH
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 7/13/22 08:00, David Hildenbrand wrote:
> On 08.07.22 21:36, Khalid Aziz wrote:
>> On 7/8/22 05:47, David Hildenbrand wrote:
>>> On 02.07.22 06:24, Andrew Morton wrote:
>>>> On Wed, 29 Jun 2022 16:53:51 -0600 Khalid Aziz <khalid.aziz@oracle.com> wrote:
>>>>
>>>>> This patch series implements a mechanism in kernel to allow
>>>>> userspace processes to opt into sharing PTEs. It adds a new
>>>>> in-memory filesystem - msharefs.
>>>>
>>>> Dumb question: why do we need a new filesystem for this?  Is it not
>>>> feasible to permit PTE sharing for mmaps of tmpfs/xfs/ext4/etc files?
>>>>
>>>
>>> IIRC, the general opinion at LSF/MM was that this approach at hand is
>>> makes people nervous and I at least am not convinced that we really want
>>> to have this upstream.
>>
>> Hi David,
> 
> Hi Khalid,
> 
>>
>> You are right that sharing page tables across processes feels scary, but at the same time threads already share PTEs and
>> this just extends that concept to processes.
> 
> They share a *mm* including a consistent virtual memory layout (VMA
> list). Page table sharing is just a side product of that. You could even
> call page tables just an implementation detail to produce that
> consistent virtual memory layout -- described for that MM via a
> different data structure.

Yes, sharing an mm and vma chain does make it different from implementation point of view.

> 
>> A number of people have commented on potential usefulness of this concept
>> and implementation.
> 
> ... and a lot of people raised concerns. Yes, page table sharing to
> reduce memory consumption/tlb misses/... is something reasonable to
> have. But that doesn't require mshare, as hugetlb has proven.
> 
> The design might be useful for a handful of corner (!) cases, but as the
> cover letter only talks about memory consumption of page tables, I'll
> not care about those. Once these corner cases are explained and deemed
> important, we might want to think of possible alternatives to explore
> the solution space.

Memory consumption by page tables is turning out to be significant issue. I mentioned one real-world example from a 
customer where a 300GB SGA on a 512GB server resulted in OOM when 1500+ processes tried to map parts of the SGA into 
their address space. Some customers are able to solve this issue by switching to hugetlbfs but that is not feasible for 
every one.

> 
>> There were concerns raised about being able to make this safe and reliable.
>> I had agreed to send a
>> second version of the patch incorporating feedback from last review and LSF/MM, and that is what v2 patch is about. The
> 
> Okay, most of the changes I saw are related to the user interface, not
> to any of the actual dirty implementation-detail concerns. And the cover
> letter is not really clear what's actually happening under the hood and
> what the (IMHO) weird semantics of the design imply (as can be seen from
> Andrews reply).

Sure, I will add more details to the cover letter next time. msharefs needs more explanation. I will highlight the 
creation of a new mm struct for mshare regions that is not owned by any process. There was another under-the-hood change 
that is listed in changelog but could have been highlighted - "Eliminated the need to point vm_mm in original vmas to 
the newly synthesized mshare mm". How the fields in new mm struct are used helped make this change and could use more 
details in cover letter.

> 
>> suggestion to extend hugetlb PMD sharing was discussed briefly. Conclusion from that discussion and earlier discussion
>> on mailing list was hugetlb PMD sharing is built with special case code in too many places in the kernel and it is
>> better to replace it with something more general purpose than build even more on it. Mike can correct me if I got that
>> wrong.
> 
> Yes, I pushed for the removal of that yet-another-hugetlb-special-stuff,
> and asked the honest question if we can just remove it and replace it by
> something generic in the future. And as I learned, we most probably
> cannot rip that out without affecting existing user space. Even
> replacing it by mshare() would degrade existing user space.
> 
> So the natural thing to reduce page table consumption (again, what this
> cover letter talks about) for user space (semi- ?)automatically for
> MAP_SHARED files is to factor out what hugetlb has, and teach generic MM
> code to cache and reuse page tables (PTE and PMD tables should be
> sufficient) where suitable.
> 
> For reasonably aligned mappings and mapping sizes, it shouldn't be too
> hard (I know, locking ...), to cache and reuse page tables attached to
> files -- similar to what hugetlb does, just in a generic way. We might
> want a mechanism to enable/disable this for specific processes and/or
> VMAs, but these are minor details.
> 
> And that could come for free for existing user space, because page
> tables, and how they are handled, would just be an implementation detail.
> 
> 
> I'd be really interested into what the major roadblocks/downsides
> file-based page table sharing has. Because I am not convinced that a
> mechanism like mshare() -- that has to be explicitly implemented+used by
> user space -- is required for that.
> 

I see two parts to what you are suggesting (please correct me if I get this wrong):

1. Implement a generic page table sharing mechanism
2. Implement a way to use this mechanism from userspace

For 1, your suggestion seems to be extract the page table sharing code from hugetlb and make it generic. My approach is 
to create a special mm struct to host the shared page tables and create a minimal set of changes to simply get PTEs from 
this special mm struct whenever a shared VMA is accessed. There may be value to extracting hugetlb page table sharing 
code and recasting it into this framework of special mm struct. I will look some more into it.

As for 2, is it fair to say you are not fond of explicit opt-in from userspace and would rather have the sharing be file 
based like hugetlb? That is worth considering but is limiting page table sharing to just file objects reasonable? A goal 
for mshare mechanism was to allow shared objects to be files, anonymous pages, RDMA buffers, whatever. Idea being if you 
can map it, you can share it with shared page tables. Maybe that is too ambitious a goal and I am open to course correction.

Thanks,
Khalid