Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Wed, 1 Sep 2021 12:32:47 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     David Hildenbrand <david@redhat.com>
Cc:     Qi Zheng <zhengqi.arch@bytedance.com>, akpm@linux-foundation.org,
        tglx@linutronix.de, hannes@cmpxchg.org, mhocko@kernel.org,
        vdavydov.dev@gmail.com, kirill.shutemov@linux.intel.com,
        mika.penttila@nextfour.com, linux-doc@vger.kernel.org,
        linux-kernel@vger.kernel.org, linux-mm@kvack.org,
        songmuchun@bytedance.com
Subject: Re: [PATCH v2 6/9] mm: free user PTE page table pages
Message-ID: <20210901153247.GJ1721383@nvidia.com>
References: <20210819031858.98043-1-zhengqi.arch@bytedance.com>
 <20210819031858.98043-7-zhengqi.arch@bytedance.com>
 <20210901135314.GA1859446@nvidia.com>
 <0c9766c9-6e8b-5445-83dc-9f2b71a76b4f@redhat.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <0c9766c9-6e8b-5445-83dc-9f2b71a76b4f@redhat.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?6s+gCerC1/OlG7erO4sz4HmBNr8whviIJ9txtjgSt8s9mtpKXjSXa1Oc9beB?=
 =?us-ascii?Q?EyB4yuVDdd2/7oAkQQ2BmJFJuCc/kHYoBeBbeXYsSpdsF1DeTlWJIUTGPbjy?=
 =?us-ascii?Q?ypLSwJeZEyD/3XEZWMeSkI7aYp0XAfF6qzjpwfwNZjqapJjs6cpkl4svcb1i?=
 =?us-ascii?Q?3Ux3gS7OSrogySs2uqbiLrmmsSeTU2/yTTALO8OxEKQpnSOaB1D3Hb7bRQIu?=
 =?us-ascii?Q?pgwdfrj2LCsKCeCTYd9G9Fb64i/A99erOn+P3ecSZ3TIMTt3qXOzaCg2qyQO?=
 =?us-ascii?Q?+16HLWb8RbcxSIBJcH+wvECgeOgOclkeE/Xo0Tn27xfgobK5r6zv/c8fLKEe?=
 =?us-ascii?Q?3y6A6+hKamdWEQth6e80pIW5QHVJbgAMEf8+oehJ5ZfdiQzsWVETIdP36orI?=
 =?us-ascii?Q?HVVIh8/lHl6ReBaPk6hQT0jzIhJL8LB1ptzXZtmA2d+RLtOFrozWVWfOTxbw?=
 =?us-ascii?Q?MNkgktBJt/Vvy1buW0K9DPWZKUM4ve7WK+ZuiJzodHBqVRONKl/q5Bw0T2uZ?=
 =?us-ascii?Q?SnvT6sKY0Z0kKbfKoLg3dH/vAGe6/Bre2KC3Gy48KJ3ZXi/xgczfYZJjbuET?=
 =?us-ascii?Q?OnjPKxgq1vEQk8nbj/OO8jytPAhDY2x0yA93k5JPXmHETUFz8DNdsaOupxm0?=
 =?us-ascii?Q?YAa33BDfgGPkekmS1CDE+dkOUlMSRk5SCVhGLcdLphyYqEFzA1t5IbaVlRMQ?=
 =?us-ascii?Q?ZDIaqEsivR+sbK5ZThpcX0h/PQ+KEQiwGwXQNkfp8xwrRunZRXaWQ8aIOdWn?=
 =?us-ascii?Q?p+mMz6h5GFGUL7hJ7eFc9doSxNkP+Enp2ivwfG+/ENj/+y4quPBt8YfEvNUP?=
 =?us-ascii?Q?bFM+Lf2VPj8PNIVVMATrZxXfvls3/Z4r3XjyPQmPcgrd1q3ievAZ4kA8PsnW?=
 =?us-ascii?Q?iUg7x2gpnR/likr4+WtrVdbpqggHppLJ0sAqEdiPOt5izbyn5OkPGAwpVzGn?=
 =?us-ascii?Q?5IAZEOCrJyjuPNHT72H45HgJaf8yt+vCcqz+Alh9XKMgKP1IVZKPXJuRvXkh?=
 =?us-ascii?Q?8TCnji1R+DSpYMi5FrD5GrHeGtJnfeRmeGnjTbhfklZaclCQM0Vmrbb4lNgB?=
 =?us-ascii?Q?6oa4nCzRgdR7KDV29AWsFx8VztZTGgQaRAJFfLKC5n/hS+K8pZDDc5QDjs4d?=
 =?us-ascii?Q?cpR6NjbiTTC+zum+/x+45RjEosZxy3vz/oB3kWtnz0WrZSZCDFFUURJFzwdl?=
 =?us-ascii?Q?u3NiddMKt6i/rBn3fud4+toCGhNSqeDozC5MhyNtrstlPgm6GLuXrFa0VbgG?=
 =?us-ascii?Q?LMkxmw4j0syk3KWPaDPo2NMl9yKEtMmLA7qxQjJ8TVEjv+I49tK+2Cpx1TRC?=
 =?us-ascii?Q?wUWtiOze3qAzHYvBnHi+4ICD?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9c01a8d9-eaa0-49df-3538-08d96d5dc095
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Sep 2021 15:32:48.8230
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: RU8MqWsHIuGqTQIK67LmGcxqyweyN1N6hDJ8GlEsXPI1Vy9f1gDWmt3bBkc/UEcM
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5111
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Sep 01, 2021 at 03:57:09PM +0200, David Hildenbrand wrote:
> On 01.09.21 15:53, Jason Gunthorpe wrote:
> > On Thu, Aug 19, 2021 at 11:18:55AM +0800, Qi Zheng wrote:
> > 
> > > diff --git a/mm/gup.c b/mm/gup.c
> > > index 2630ed1bb4f4..30757f3b176c 100644
> > > +++ b/mm/gup.c
> > > @@ -500,6 +500,9 @@ static struct page *follow_page_pte(struct vm_area_struct *vma,
> > >   	if (unlikely(pmd_bad(*pmd)))
> > >   		return no_page_table(vma, flags);
> > > +	if (!pte_try_get(mm, pmd))
> > > +		return no_page_table(vma, flags);
> > > +
> > >   	ptep = pte_offset_map_lock(mm, pmd, address, &ptl);
> > 
> > This is not good on a performance path, the pte_try_get() is
> > locking/locking the same lock that pte_offset_map_lock() is getting.
> 
> Yes, and we really need patch #8, anything else is just confusing reviewers.

It is a bit better with patch 8, but it is still not optimal, we don't
need to do the atomic work at all if the entire ptep is accessed while
locked. So the above is stil not what I would expect here, even with
RCU.

eg I would expect that this kind of change would work first with the
existing paired acessors, ie

	pte = pte_offset_map(pmd, address);
	pte_unmap(pte);

Should handle the refcount under the covers, and same kind of idea for
the _locked/_unlocked varient.

Only places that don't already use that pairing should get modified.

To do this we have to extend the API so that pte_offset_map() can
fail, or very cleverly return some kind of global non-present pte page
(I wonder if the zero page would work?)

> > Also, I don't really understand how this scheme works with
> > get_user_pages_fast.
> 
> With the RCU change it in #8 it should work just fine, because RCU
> synchronize has to wait either until all other CPUs have left the RCU read
> section, or re-enabled interrupts.

So at this point in the series fast gup is broken, that does mean the
series presentation really needs to be reworked. The better
presentation is to add the API changes, with a
no-functional-difference implementation, push the new API in well
split patches to all the consumption sites, then change the API to
have the new semantics.

RCU and refcount to free the page levels seems like a reasonable
approach, but I have to say I haven't thought it through fully - are
all the contexts that have the pte deref safe to do call_rcu?

Jason