Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 29 Dec 2022 23:22:28 +0900
From:   Hyeonggon Yoo <42.hyeyoo@gmail.com>
To:     Matthew Wilcox <willy@infradead.org>
Cc:     linux-mm@kvack.org, liam.howlett@oracle.com, surenb@google.com,
        ldufour@linux.ibm.com, michel@lespinasse.org, vbabka@suse.cz,
        linux-kernel@vger.kernel.org
Subject: Re: [QUESTION] about the maple tree and current status of mmap_lock
 scalability
Message-ID: <Y62ipKlWGEbJZKXv@hyeyoo>
References: <EC51CFA7-2BC8-4F72-A7D4-3B1A778EDB37@gmail.com>
 <Y6ysHNPvKayTfeq8@casper.infradead.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Y6ysHNPvKayTfeq8@casper.infradead.org>
Precedence: bulk

On Wed, Dec 28, 2022 at 08:50:36PM +0000, Matthew Wilcox wrote:
> On Wed, Dec 28, 2022 at 09:48:51PM +0900, Hyeonggon Yoo wrote:
> > Hello mm folks,
> > 
> > I have a few questions about the current status of mmap_lock scalability.
> > 
> > =============================================================
> > What is currently causing the kernel to use mmap_lock to protect the maple tree?
> > =============================================================
> > 
> > I understand that the long-term goal is to remove the need for mmap_lock in readers
> > while traversing the maple tree, using techniques such as RCU or SPF.
> > What is the biggest obstacle preventing this from being achieved at this time?
> 
> The long term goal is even larger than this.  Ideally, the VMA tree
> would be protected by a spinlock rather than a mutex. 

You mean replacing mmap_lock rwsem with a spinlock?
How is that possible if readers can take it for page fault?

> That turned out
> to be too large a change for the moment (and isn't all that important
> compared to enabling RCU readers)

Yeah, better to take one step at a time.

> 
> > ==================================================
> > How does the maple tree provide RCU-safe manipulation of VMAs?
> > ==================================================
> > 
> > Is it similar to the approach suggested in the RCUVM paper (replacing the original
> > root node with a new root node that shares most of its nodes and deferring
> > the freeing of stale nodes using RCU)?
> > 
> > I'm having difficulty understanding the design of the maple tree in this regard.
> > 
> > [RCUVM paper] https://pdos.csail.mit.edu/papers/rcuvm:asplos12.pdf
> 
> While I've read the RCUVM paper, I wouldn't say it was particularly an
> inspiration.  The Maple Tree is independent of the VM; it's a general
> purpose B-tree.

My intention was to ask how to synchronize with other VMA operations
after the tree traversal with RCU. (Because it's unreasonable to handle
page fault in RCU read-side critical section)

Per-VMA lock seem to solve it by taking the VMA lock in read mode within
RCU read-side critical section.

> As with any B-tree, when modifying a node, we don't
> touch nodes that we don't need to touch.  As with any RCU data structure,
> we defer freeing things while RCU readers might still have a reference
> to them.
> 
> We don't necessarily go all the way to the root node when modifying a
> leaf node.  For example, if we have this structure:
> 
> Root: Node A, 4000, Node B
> Node A: p1, 50, p2, 100, p3, 150, p4, 200, NULL, 250, p6, 1000, p7
> Node B: p8, 4050, p9, 4100, p10, 4150, p11, 4200, NULL, 4250, p13
> 
> and we replace p4 with a NULL over the whole range from 150-199,
> we construct a new Node A2 that contains:
> 
> Node A2: p1, 50, p2, 100, p3, 150, NULL, 250, p6, 1000, p7
> 
> and we simply write A2 over the entry in Root.  Then we mark Node A as
> dead and RCU-free Node A.  There's no need to replace Root as stores
> to a pointer are atomic.

Thank you for explaining things in an easy and intuitive way.
Okay, I get it's not a big problem to update the value(s) in a
B-tree in RCU-safe way.

> If we need to rebalance between Node A and
> Node B, we will need to create a new Root (as well as both A and B),
> mark all of them as dead and RCU-free them.

-- 
Thanks,
Hyeonggon