This is a patchset implementing MMU notifier callbacks based on Andrea's
earlier work. These are needed if Linux pages are referenced from something
else than tracked by the rmaps of the kernel (an external MMU).
The known immediate users are
KVM
- Establishes a refcount to the page via get_user_pages().
- External references are called spte.
- Has page tables to track pages whose refcount was elevated(?) but
no reverse maps.
GRU
- Simple additional hardware TLB (possibly covering multiple instances of
Linux)
- Needs TLB shootdown when the VM unmaps pages.
- Determines page address via follow_page (from interrupt context) but can
fall back to get_user_pages().
- No page reference possible since no page status is kept..
XPmem
- Allows use of a processes memory by remote instances of Linux.
- Provides its own reverse mappings to track remote pte.
- Established refcounts on the exported pages.
- Must sleep in order to wait for remote acks of ptes that are being
cleared.
Known issues:
- RCU quiescent periods are required on registering
notifiers to guarantee visibility to other processors.
Andrea's mmu_notifier #4 -> RFC V1
- Merge subsystem rmap based with Linux rmap based approach
- Move Linux rmap based notifiers out of macro
- Try to account for what locks are held while the notifiers are
called.
- Develop a patch sequence that separates out the different types of
hooks so that we can review their use.
- Avoid adding include to linux/mm_types.h
- Integrate RCU logic suggested by Peter.
V1->V2:
- Improve RCU support
- Use mmap_sem for mmu_notifier register / unregister
- Drop invalidate_page from COW, mm/fremap.c and mm/rmap.c since we
already have invalidate_range() callbacks there.
- Clean compile for !MMU_NOTIFIER
- Isolate filemap_xip strangeness into its own diff
- Pass a the flag to invalidate_range to indicate if a spinlock
is held.
- Add invalidate_all()
V2->V3:
- Further RCU fixes
- Fixes from Andrea to fixup aging and move invalidate_range() in do_wp_page
and sys_remap_file_pages() after the pte clearing.
V3->V4:
- Drop locking and synchronize_rcu() on ->release since we know on release that
we are the only executing thread. This is also true for invalidate_all() so
we could drop off the mmu_notifier there early. Use hlist_del_init instead
of hlist_del_rcu.
- Do the invalidation as begin/end pairs with the requirement that the driver
holds off new references in between.
- Fixup filemap_xip.c
- Figure out a potential way in which XPmem can deal with locks that are held.
- Robin's patches to make the mmu_notifier logic manage the PageRmapExported bit.
- Strip cc list down a bit.
- Drop Peters new rcu list macro
- Add description to the core patch
V4->V5:
- Provide missing callouts for mremap.
- Provide missing callouts for copy_page_range.
- Reduce mm_struct space to zero if !MMU_NOTIFIER by #ifdeffing out
structure contents.
- Get rid of the invalidate_all() callback by moving ->release in place
of invalidate_all.
- Require holding mmap_sem on register/unregister instead of acquiring it
ourselves. In some contexts where we want to register/unregister we are
already holding mmap_sem.
- Split out the rmap support patch so that there is no need to apply
all patches for KVM and GRU.
--
With this set of patches, I think we have enough to get xpmem working
with most types of mappings. In the past, we operated without any of
these callouts by significantly restricting why types of mappings could
remotely fault and what types of operations the user could do. With
this set, I am certain we can continue to meet the above assumptions.
That said, I would like to discuss __xip_unmap in more detail.
Currently, it is calling mmu_notifier _begin and _end under the
i_mmap_lock. I _THINK_ the following will make it so we could support
__xip_unmap (although I don't recall ever seeing that done on ia64 and
don't even know what the circumstances are for its use).
Thanks,
Robin
Index: mmu_notifiers-cl-v5/mm/filemap_xip.c
===================================================================
--- mmu_notifiers-cl-v5.orig/mm/filemap_xip.c 2008-02-01 05:38:32.000000000 -0600
+++ mmu_notifiers-cl-v5/mm/filemap_xip.c 2008-02-01 05:39:08.000000000 -0600
@@ -184,6 +184,7 @@ __xip_unmap (struct address_space * mapp
if (!page)
return;
+ mmu_rmap_notifier(invalidate_page, page);
spin_lock(&mapping->i_mmap_lock);
vma_prio_tree_foreach(vma, &iter, &mapping->i_mmap, pgoff, pgoff) {
mm = vma->vm_mm;
Argh,
Here is the correct one. Sorry
On Fri, Feb 01, 2008 at 05:58:41AM -0600, Robin Holt wrote:
> With this set of patches, I think we have enough to get xpmem working
> with most types of mappings. In the past, we operated without any of
> these callouts by significantly restricting why types of mappings could
> remotely fault and what types of operations the user could do. With
> this set, I am certain we can continue to meet the above assumptions.
>
> That said, I would like to discuss __xip_unmap in more detail.
>
> Currently, it is calling mmu_notifier _begin and _end under the
> i_mmap_lock. I _THINK_ the following will make it so we could support
> __xip_unmap (although I don't recall ever seeing that done on ia64 and
> don't even know what the circumstances are for its use).
Index: mmu_notifiers-cl-v5/mm/filemap_xip.c
===================================================================
--- mmu_notifiers-cl-v5.orig/mm/filemap_xip.c 2008-02-01 05:38:32.000000000 -0600
+++ mmu_notifiers-cl-v5/mm/filemap_xip.c 2008-02-01 06:09:09.000000000 -0600
@@ -184,6 +184,10 @@ __xip_unmap (struct address_space * mapp
if (!page)
return;
+ if (unlikely(PageExternalRmap(page))) {
+ mmu_rmap_notifier(invalidate_page, page);
+ ClearPageExternalRmap(page);
+ }
spin_lock(&mapping->i_mmap_lock);
vma_prio_tree_foreach(vma, &iter, &mapping->i_mmap, pgoff, pgoff) {
mm = vma->vm_mm;
On Fri, 1 Feb 2008, Robin Holt wrote:
> Currently, it is calling mmu_notifier _begin and _end under the
> i_mmap_lock. I _THINK_ the following will make it so we could support
> __xip_unmap (although I don't recall ever seeing that done on ia64 and
> don't even know what the circumstances are for its use).
Its called under lock yes.
The problem with this fix is that we currently have the requirement that
the rmap invalidate_all call requires the pagelock to be held. That is not
the case here. So I used _begin/_end to skirt the issue.
If you do not need the Pagelock to be held (it holds off modifications on
the page!) then we are fine.
On Thu, Jan 31, 2008 at 09:04:39PM -0800, Christoph Lameter wrote:
> - Has page tables to track pages whose refcount was elevated(?) but
> no reverse maps.
Just a correction, rmaps exists or swap couldn't be sane, it's just
that it's not built on the page_t because the guest memory is really
virtual and not physical at all (hence it swaps really well, thanks to
the regular linux VM algorithms without requiring any KVM knowledge at
all, it all looks (shared) anonymous memory as far as linux is
concerned ;).
Great news! I have taken over Dean's xpmem patch set while he is on
sabbatical. Before he left, he had his patch mostly working on top of
this patch set. We had one deadlock. I have coded for that specific
deadlock and xpmem now passes a simple grant/attach/fault/fork/unmap/map
test.
After analyzing it, I believe we still have a nearly related deadlock
which will require some refactoring of code. I am certain that the
same mechanism I used for this deadlock break will work in that case,
but it will require too many changes for me to finish this weekend.
For our customer base, this case, in the past, has resulted in termination
of the application and our MPI library specifically states that this
mode of operation is not permitted, so I think we will be able to pass
their regression tests. I will need to coordinate that early next week.
The good news, at this point, Christoph's version 5 of the mmu_notifiers
appears to work for xpmem. The mmu_notifier call-outs where the
in_atomic flag is set still result in a BUG_ON. That is not an issue
for our normal customer as our MPI already states this is not a valid
mode of operation and provides means to avoid those types of mappings.
Thanks,
Robin