Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 7 Jun 2023 13:53:10 -0700
From:   Mike Kravetz <mike.kravetz@oracle.com>
To:     Andrew Morton <akpm@linux-foundation.org>
Cc:     Gerd Hoffmann <kraxel@redhat.com>,
        Junxiao Chang <junxiao.chang@intel.com>,
        kirill.shutemov@linux.intel.com, mhocko@suse.com,
        jmarchan@redhat.com, linux-mm@kvack.org,
        linux-kernel@vger.kernel.org, muchun.song@linux.dev,
        Vivek Kasireddy <vivek.kasireddy@intel.com>,
        Dongwon Kim <dongwon.kim@intel.com>,
        James Houghton <jthoughton@google.com>,
        dri-devel@lists.freedesktop.org
Subject: Re: [PATCH] mm: fix hugetlb page unmap count balance issue
Message-ID: <20230607205310.GA4122@monkey>
References: <20230512072036.1027784-1-junxiao.chang@intel.com>
 <CADrL8HV25JyeaT=peaR7NWhUiaBz8LzpyFosYZ3_0ACt+twU6w@mail.gmail.com>
 <20230512232947.GA3927@monkey>
 <20230515170259.GA3848@monkey>
 <20230516223440.GA30624@monkey>
 <20230607120312.6da5cea7677ec1a3da35b92c@linux-foundation.org>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <20230607120312.6da5cea7677ec1a3da35b92c@linux-foundation.org>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Sml0YWJsMHBEVnJWbVNseTBXOGg2d2N5Q2g2dXNKQ0hDWjRJandtR3pWQ3Jo?=
 =?utf-8?B?MTJYaUQzYVdVQlh6VUZ5eHAvSklCYS9qUmZwT3hQa0xydkplT3dWY3ZkbHBl?=
 =?utf-8?B?Y21mRjhRQlc2Uk04MWlzd2V6UDVuUXNOL2xpRkRrVG11SzRmci9jQWs5YXZB?=
 =?utf-8?B?ai9jSTk3Mk9lQVBXUVdVc1hudElXUEkveGJEZmpKR202SmhyZnZNdDBHTjgy?=
 =?utf-8?B?VHRiZzROQ3VnMWllcnZvWnB2NTFLTzR2TmhLOEEzUmJ0RkhqeUM2dVl2eERO?=
 =?utf-8?B?RnMrclRlUHkxTFNEZ2tQLytrRWszM1dVMGNkS1RGRUtWNkdLWW1SSDJvWmxa?=
 =?utf-8?B?YzJVbUltaXNTREY2SFlVSWZXZ0hzaDRQZ2haMWFYamlWdmxRTjNyaVBiTnd6?=
 =?utf-8?B?N2RxampjaTIvSXJsK3dGWFQxTkpCS1dsMmpjRkNDUy8vbXRZcWt0d0tONWZE?=
 =?utf-8?B?NDlrUGRKVlRqMURKUW8yR1MwMUNaUGlGUkZjMkhacjNrdGF1WHdFelNtRjJW?=
 =?utf-8?B?RjNhdjMraGZobFNDcnpLRFcwSWdkSmFkdXRuY1U5MERzZ0YwWTZ3WWNUYnU1?=
 =?utf-8?B?WHRMV1RoS0RIV1N1VndXajZPaEpJSEh6VTRlV044cFBOTmR1MHNtOGVobktT?=
 =?utf-8?B?L0x4Z092RnNacElWSjdldDNnK3hWc1lRVTZDVWYzbU9EcVpLdU5lU0ZBcDQ5?=
 =?utf-8?B?bkVIcGhTcVFkeEdXbTBMeU16czJsZzRuejFMWmxkbWN5MXB5eEFEYlBlQ09J?=
 =?utf-8?B?dzNDTnFkbmlHR0Q3aEpSU3BXUENQdXBnRzlCM2psZjNSZThES3QvcFhZTnJG?=
 =?utf-8?B?UWJsd0dncGFiTHZqRExtMDRuSU92Qk9YVlZ2dXo2VTc1Tmo4UVB0M3RyVFlS?=
 =?utf-8?B?b3B2M2w3QkdyakdXUFo0aVdTT2NtcHVoWFAxa3FudFJHTkNoVytDOTNrMENK?=
 =?utf-8?B?NXVOWDNPQnUwa2lLaEUxajlJVVZYREZmTzMyL283M1RvN2ZDZE0xY0RtOWZj?=
 =?utf-8?B?QU5XNkgra045WU9yU2tWRG4wNHNZUEV1amVZRUdyZklTTEFUeXdYVzc1VUJ0?=
 =?utf-8?B?dnNoRjV4TSsyVlZGblphRll4U0hBWjFPMzhxWGpGWk9pKzd5MVhwZkwrR202?=
 =?utf-8?B?b1dXQ01PMC9nVjBPbW1oY3ZyaUpXQmVUQ0JRSTFBbk5VVitzSy8ySzRLRTZV?=
 =?utf-8?B?d05LSlhBUzJiK2VWRlRiejQ4RkgrbS9EcmlOd1hmMEJUMmlsMk53UzYybzlQ?=
 =?utf-8?B?aGVYL21ZdDBYSXd4aEM3WUQ2QzhzVERWbFJWeHE0ZlZPMDJwZlYxRGNQMDJ0?=
 =?utf-8?B?dmhDV2ZmWWVxeXN0aDZTRlBjRmdHUlRMS1JTYllueitpRHZEKytXVzh6cGVs?=
 =?utf-8?B?dHVYek9qRkNyVDNkdmFBTWhtWnNkajZzYlRKOFJsRldRcS9JelhMMjJydk1v?=
 =?utf-8?B?SFQzTWhmM0I4UHpkeGF1MFk1emRWVlRRK3cyQllJUnhrRGtpRmpoUW5XaE41?=
 =?utf-8?B?Y3lrZnFHcU1pYUl6dk5XRmNNNkRQNDBmekR4aEhnNGtMaVFYOGNzYkpzR1pk?=
 =?utf-8?B?Q3psTHNXdHhFWXE1RXBrM3FRTjd5YmZtMXZnbkh1V00vR2krWmU5YnpuN2lG?=
 =?utf-8?B?T05XMHdBWUJEcU5YRVlTR0ZFcVhDTnhQZ1ptbEI1YjhoaDBCcUszVkE1ZENS?=
 =?utf-8?B?VS9WWkU1VFZYYnRTeDIvVndXbmJZa0ZoaUt5OFM4UTNFcmo3akxITmtsYnpC?=
 =?utf-8?B?N1BCcXNzMThibDVpMDR0OVRjUjhZQUUvQ1Zzem05Y1pYOFdqY05XQUJPRTZo?=
 =?utf-8?B?ekNIMnU3NW90MVlOcW5zaEdEQ3oyOWRzOWRGWDE3Vm1MQzg0cHNRK1ovVy9q?=
 =?utf-8?B?d1ZWOFpBV2ErWjl0RENWa2xBc1MwYUFLMVBuQk5BaDRWNzQ3V0g4RzUvSmV6?=
 =?utf-8?B?ektiVXBidWhUdWYxWTR3aXllMnk2ajg2ZVozVkVic0wrQ1Y5bjVCRlRWTC9J?=
 =?utf-8?B?OUROMVFyaXNvK3ZWTDBibGEwSXowdHJTNVRWK1AwYnd4Ykc5VWUwRUdMNUZl?=
 =?utf-8?B?cU9Gekdzckx1NFRuYSswVUNhYzRrNDAvdnZtSDdNNFVwOEM1V254VUtWcEpq?=
 =?utf-8?Q?N2ez0T8SWu/w/1kb8kCU/J0/L?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?utf-8?B?Y1JCOC8rNW1KOEZtMHljajRlQ1hrSy9kZk1yNFZuZUlseUxxR1dmTnZ4dmY5?=
 =?utf-8?B?RXZyS1hmdGtmZ1l3WnhYU2ZvMkpmeGQ0NDlFeVI5WkdlSkJxcS9zdXFhYzlv?=
 =?utf-8?B?THZuQUhhbkwrSloyRzhKVm9rUFNYUTJ5Z0REQ05FcngvbThBekg4a2FIbWFM?=
 =?utf-8?B?QUEzZ2pzT3hhNGxaMXN5ZXJnaktxM3Z4bng4eE0zUlljNHU3YlJhSFJFNmti?=
 =?utf-8?B?SjVTTmFWRFM5Q1NrOFlUMmJGcVlKaWNBUVhLeG5MQTNuZnJjNEc4Yi81dHNO?=
 =?utf-8?B?M2lPQTdRdHVBOXZkZ25zN0FCYUx1NHBYTmN4aXY3QzRHTnF1cnFSamgxZWpQ?=
 =?utf-8?B?RFI1NUpjcHVNNnBVVGJ5U01NaDdQZFJ2d2lLRjZsNXhkZzBMTlIrTEdYQmZh?=
 =?utf-8?B?RVhkNG1ZWnN2bS80OS8wZ3dVWC9Jd3BKYWlmZWQzTzRHWmJjN2tMcWNOaEFI?=
 =?utf-8?B?VXhBTFQrYU1PQWFWaWlFaWx3T21jejgyRUtyb01waTVNbFJub2owdWNpQUpO?=
 =?utf-8?B?UUJ1Z1FYSWNwWTd2a2E0NUZuMFNVaUppdy82Q0RSMzV3Smt3ZGZyNlZaN3Fa?=
 =?utf-8?B?enEzTEk3bjQwN0RVZ0IvYW5OY2VoZFhYOTZMbzJvZkdvbVRMaUhxcndPL2Ja?=
 =?utf-8?B?MUVYNzAwSXhMVGh5bjIxQVBnWjlQS0tVcXB1UmlBblJBZEZjSzFjWjMvY1ND?=
 =?utf-8?B?OENiOFVhd1NNL0k3UFlrSVNtTDBka2FFNmk3UUZXMjJYdWJYa2VBbHczekxw?=
 =?utf-8?B?T0xrbjVqNVh0WEI2S2xqUDhMZWVWSzVWakpTcWR4WHJaeWNybzRyQkJWdFdR?=
 =?utf-8?B?S0ZzV1NEN0lsZWVuYS8zMm5ENzAwMWdPTWI4VmozejJXeUphSjVIc1ZtTlhv?=
 =?utf-8?B?dnk0MzZNdFN2Y3NNNEY0K3ZVSlR0Z2pyTDZvYUtNZVVlNmVVSTNWYWdnRGF3?=
 =?utf-8?B?TGdVQ1pmUFp4c2krbzUweXQyeTcwNk9GSCs4SWI5UzVkOE1CS3J4d2U3L1VH?=
 =?utf-8?B?SHBGbWVZVlN5eCtJc1JDTytaTWtMdDd6QmdjZURUUUlQSUVhR1Ric2VyVkY5?=
 =?utf-8?B?T2Z6TFVuc1dlODVLcU1PTWh5UnR2dEZkUG0yaHJ5MnJwSWdVUEhqeEZnZHhS?=
 =?utf-8?B?aGUwNk81SmFnL1BBVlVEK0hLbmVKS01HUlBBR2RxRW0rVFFtdldkcFlyV3JO?=
 =?utf-8?B?T28zV01NTFZGbW9rTDJBMTVnSlVkWTlPMFkyRGd1aThDNGJXUXRPVk1ORWMx?=
 =?utf-8?B?QzNTd0dpTHhCc1paSTdqWFUxcmllbUt2dG1KcjVKeVp6TmZXNzFjb2ZOSGlI?=
 =?utf-8?B?Y1h5NmJqOFFDSWkxdUlHY1ZGZ0VsZ1VjcUphME9ERjdGS2FBekgwMnpRc201?=
 =?utf-8?B?MWFDcnZ0eUdpbE0vb2FJWUU5QkltemNjRDlQa1phcllNTDhWc1p1aUhkMTZW?=
 =?utf-8?B?U2Z6U1JYemJCODVHRkl5b0YvS09xalhaNVhydktBPT0=?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4e5fa812-31b0-468b-aa6e-08db67993652
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Jun 2023 20:53:14.4283
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: GdCH0HhUiFx0yXczPCgZ8KinhT/Tw07G4hHjlxuyxBfOBpUdF3mLQ889QKANixLuLwTcyiaGhazhShTHxuvFMw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR10MB7003
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.573,FMLib:17.11.176.26
 definitions=2023-06-07_11,2023-06-07_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxlogscore=999 bulkscore=0
 suspectscore=0 phishscore=0 adultscore=0 malwarescore=0 mlxscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000
 definitions=main-2306070181
X-Proofpoint-GUID: 6ZTmBt3e6C5d4jYZU7Gxi0wudqFqrOWL
X-Proofpoint-ORIG-GUID: 6ZTmBt3e6C5d4jYZU7Gxi0wudqFqrOWL
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 06/07/23 12:03, Andrew Morton wrote:
> On Tue, 16 May 2023 15:34:40 -0700 Mike Kravetz <mike.kravetz@oracle.com> wrote:
> 
> > On 05/15/23 10:04, Mike Kravetz wrote:
> > > On 05/12/23 16:29, Mike Kravetz wrote:
> > > > On 05/12/23 14:26, James Houghton wrote:
> > > > > On Fri, May 12, 2023 at 12:20 AM Junxiao Chang <junxiao.chang@intel.com> wrote:
> > > > > 
> > > > > This alone doesn't fix mapcounting for PTE-mapped HugeTLB pages. You
> > > > > need something like [1]. I can resend it if that's what we should be
> > > > > doing, but this mapcounting scheme doesn't work when the page structs
> > > > > have been freed.
> > > > > 
> > > > > It seems like it was a mistake to include support for hugetlb memfds in udmabuf.
> > > > 
> > > > IIUC, it was added with commit 16c243e99d33 udmabuf: Add support for mapping
> > > > hugepages (v4).  Looks like it was never sent to linux-mm?  That is unfortunate
> > > > as hugetlb vmemmap freeing went in at about the same time.  And, as you have
> > > > noted udmabuf will not work if hugetlb vmemmap freeing is enabled.
> > > > 
> > > > Sigh!
> > > > 
> > > > Trying to think of a way forward.
> > > > -- 
> > > > Mike Kravetz
> > > > 
> > > > > 
> > > > > [1]: https://lore.kernel.org/linux-mm/20230306230004.1387007-2-jthoughton@google.com/
> > > > > 
> > > > > - James
> > > 
> > > Adding people and list on Cc: involved with commit 16c243e99d33.
> > > 
> > > There are several issues with trying to map tail pages of hugetllb pages
> > > not taken into account with udmabuf.  James spent quite a bit of time trying
> > > to understand and address all the issues with the HGM code.  While using
> > > the scheme proposed by James, may be an approach to the mapcount issue there
> > > are also other issues that need attention.  For example, I do not see how
> > > the fault code checks the state of the hugetlb page (such as poison) as none
> > > of that state is carried in tail pages.
> > > 
> > > The more I think about it, the more I think udmabuf should treat hugetlb
> > > pages as hugetlb pages.  They should be mapped at the appropriate level
> > > in the page table.  Of course, this would impose new restrictions on the
> > > API (mmap and ioctl) that may break existing users.  I have no idea how
> > > extensively udmabuf is being used with hugetlb mappings.
> > 
> > Verified that using udmabug on a hugetlb mapping with vmemmap optimization will
> > BUG as:
> 
> BUGs aren't good.  Can we please find a way to push this along?
> 
> Have we heard anything from any udmabuf people?
> 

I have not heard anything.  When this issue popped up, it took me by surprise.

udmabuf maintainer (Gerd Hoffmann), the people who added hugetlb support and
the list where udmabuf was developed (dri-devel@lists.freedesktop.org) have
been on cc.

My 'gut reaction' would be to remove hugetlb support from udmabuf.  From a
quick look, if we really want this support then there will need to be some
API changes.  For example UDMABUF_CREATE should be hugetlb page aligned
and a multiple of hugetlb page size if using a hugetlb mapping.

It would be good to know about users of the driver.
-- 
Mike Kravetz