Hi Linus,
Please consider pulling the XArray patch set. The XArray provides an
improved interface to the radix tree data structure, providing locking
as part of the API, specifying GFP flags at allocation time, eliminating
preloading, less re-walking the tree, more efficient iterations and not
exposing RCU-protected pointers to its users.
This patch set introduces the XArray implementation and converts the
pagecache over to use it. The page cache is the most complex and
important user of the radix tree, so converting it was most important.
I have followup patches to convert the other users of the radix tree
over to the XArray, but that'll be another hundred or so patches and I
want to get this part in first.
There are two conflicts I wanted to flag; the first is against the
linux-nvdimm tree. I rebased on top of one of the branches that went
into that tree, so if you pull my tree before linux-nvdimm, you'll get
fifteen commits I've had no involvement with.
The other is a fairly trivial one with ext4. I've incorporated the
change from Ross that Ted merged into that tree into my tree, so please
take the resolution in my tree.
----------------------------------------------------------------
The following changes since commit 0c3a2a2ae2d76df077cf5a3c36ab8ac700058447:
libnvdimm, pmem: Restore page attributes when clearing errors (2018-07-24 08:53:37 -0700)
are available in the Git repository at:
git://git.infradead.org/users/willy/linux-dax.git xarray
for you to fetch changes up to 8d7cf3e16ed4a12ab6bfc05ccb60d70cb6200ffb:
radix tree: Remove radix_tree_clear_tags (2018-08-10 00:49:25 -0400)
----------------------------------------------------------------
Matthew Wilcox (75):
Update email address
radix tree test suite: Enable ubsan
dax: Fix use of zero page
idr: Permit any valid kernel pointer to be stored
xarray: Replace exceptional entries
xarray: Change definition of sibling entries
xarray: Add definition of struct xarray
xarray: Define struct xa_node
xarray: Add documentation
xarray: Add XArray load operation
xarray: Add XArray tags
xarray: Add XArray unconditional store operations
xarray: Add XArray conditional store operations
xarray: Add XArray iterators
xarray: Extract entries from an XArray
xarray: Destroy an XArray
xarray: Step through an XArray
xarray: Add xas_for_each_conflict
xarray: Add xas_create_range
xarray: Add MAINTAINERS entry
page cache: Rearrange address_space
page cache: Convert hole search to XArray
page cache: Add and replace pages using the XArray
page cache: Convert page deletion to XArray
page cache: Convert find_get_entry to XArray
page cache: Convert find_get_entries to XArray
page cache: Convert find_get_pages_range to XArray
page cache: Convert find_get_pages_contig to XArray
page cache; Convert find_get_pages_range_tag to XArray
page cache: Convert find_get_entries_tag to XArray
page cache: Convert filemap_map_pages to XArray
radix tree test suite: Convert regression1 to XArray
page cache: Convert delete_batch to XArray
page cache: Remove stray radix comment
page cache: Convert filemap_range_has_page to XArray
mm: Convert page-writeback to XArray
mm: Convert workingset to XArray
mm: Convert truncate to XArray
mm: Convert add_to_swap_cache to XArray
mm: Convert delete_from_swap_cache to XArray
mm: Convert __do_page_cache_readahead to XArray
mm: Convert page migration to XArray
mm: Convert huge_memory to XArray
mm: Convert collapse_shmem to XArray
mm: Convert khugepaged_scan_shmem to XArray
mm: Convert is_page_cache_freeable to XArray
pagevec: Use xa_tag_t
shmem: Convert shmem_radix_tree_replace to XArray
shmem: Convert shmem_confirm_swap to XArray
shmem: Convert find_swap_entry to XArray
shmem: Convert shmem_add_to_page_cache to XArray
shmem: Convert shmem_alloc_hugepage to XArray
shmem: Convert shmem_free_swap to XArray
shmem: Convert shmem_partial_swap_usage to XArray
memfd: Convert memfd_wait_for_pins to XArray
memfd: Convert memfd_tag_pins to XArray
shmem: Comment fixups
btrfs: Convert page cache to XArray
fs: Convert buffer to XArray
fs: Convert writeback to XArray
nilfs2: Convert to XArray
f2fs: Convert to XArray
dax: Rename some functions
dax: Hash on XArray instead of mapping
dax: Convert dax_insert_pfn_mkwrite to XArray
dax: Convert dax_layout_busy_page to XArray
dax: Convert __dax_invalidate_entry to XArray
dax: Convert dax writeback to XArray
dax: Convert dax_lock_mapping_entry to XArray
dax: Convert page fault handlers to XArray
page cache: Finish XArray conversion
radix tree: Remove radix_tree_update_node_t
radix tree: Remove split/join code
radix tree: Remove radix_tree_maybe_preload_order
radix tree: Remove radix_tree_clear_tags
.clang-format | 1 -
.mailmap | 7 +
Documentation/core-api/index.rst | 1 +
Documentation/core-api/xarray.rst | 392 ++++++
MAINTAINERS | 17 +-
arch/powerpc/include/asm/book3s/64/pgtable.h | 4 +-
arch/powerpc/include/asm/nohash/64/pgtable.h | 4 +-
drivers/gpu/drm/i915/i915_gem.c | 17 +-
fs/btrfs/compression.c | 6 +-
fs/btrfs/extent_io.c | 12 +-
fs/buffer.c | 14 +-
fs/dax.c | 910 ++++++-------
fs/ext4/inode.c | 2 +-
fs/f2fs/data.c | 6 +-
fs/f2fs/dir.c | 2 +-
fs/f2fs/f2fs.h | 2 +-
fs/f2fs/inline.c | 2 +-
fs/f2fs/node.c | 6 +-
fs/fs-writeback.c | 25 +-
fs/gfs2/aops.c | 2 +-
fs/inode.c | 2 +-
fs/nfs/blocklayout/blocklayout.c | 2 +-
fs/nilfs2/btnode.c | 26 +-
fs/nilfs2/page.c | 29 +-
fs/proc/task_mmu.c | 2 +-
include/linux/fs.h | 63 +-
include/linux/pagemap.h | 10 +-
include/linux/pagevec.h | 8 +-
include/linux/radix-tree.h | 136 +-
include/linux/swap.h | 22 +-
include/linux/swapops.h | 19 +-
include/linux/xarray.h | 1054 ++++++++++++++-
lib/Kconfig | 7 +
lib/Kconfig.debug | 3 +
lib/Makefile | 3 +-
lib/idr.c | 70 +-
lib/radix-tree.c | 592 ++------
lib/test_xarray.c | 837 ++++++++++++
lib/xarray.c | 1784 +++++++++++++++++++++++++
mm/filemap.c | 724 +++++-----
mm/huge_memory.c | 17 +-
mm/khugepaged.c | 178 +--
mm/madvise.c | 2 +-
mm/memcontrol.c | 2 +-
mm/memfd.c | 105 +-
mm/migrate.c | 48 +-
mm/mincore.c | 2 +-
mm/page-writeback.c | 72 +-
mm/readahead.c | 10 +-
mm/shmem.c | 193 +--
mm/swap.c | 6 +-
mm/swap_state.c | 119 +-
mm/truncate.c | 27 +-
mm/vmscan.c | 10 +-
mm/workingset.c | 70 +-
tools/include/asm-generic/bitops.h | 1 +
tools/include/asm-generic/bitops/atomic.h | 9 -
tools/include/asm-generic/bitops/non-atomic.h | 109 ++
tools/include/linux/bitmap.h | 1 +
tools/include/linux/kernel.h | 1 +
tools/include/linux/spinlock.h | 12 +-
tools/testing/radix-tree/.gitignore | 1 +
tools/testing/radix-tree/Makefile | 16 +-
tools/testing/radix-tree/benchmark.c | 91 --
tools/testing/radix-tree/bitmap.c | 23 +
tools/testing/radix-tree/generated/autoconf.h | 1 +
tools/testing/radix-tree/idr-test.c | 69 +-
tools/testing/radix-tree/linux/bug.h | 1 +
tools/testing/radix-tree/linux/kconfig.h | 1 +
tools/testing/radix-tree/linux/kernel.h | 5 +
tools/testing/radix-tree/linux/lockdep.h | 11 +
tools/testing/radix-tree/linux/radix-tree.h | 1 -
tools/testing/radix-tree/linux/rcupdate.h | 2 +
tools/testing/radix-tree/linux/xarray.h | 2 +
tools/testing/radix-tree/main.c | 21 +-
tools/testing/radix-tree/multiorder.c | 272 +---
tools/testing/radix-tree/regression1.c | 58 +-
tools/testing/radix-tree/regression3.c | 23 -
tools/testing/radix-tree/tag_check.c | 29 -
tools/testing/radix-tree/test.c | 8 +-
tools/testing/radix-tree/test.h | 1 +
tools/testing/radix-tree/xarray.c | 35 +
82 files changed, 5749 insertions(+), 2741 deletions(-)
create mode 100644 Documentation/core-api/xarray.rst
create mode 100644 lib/test_xarray.c
create mode 100644 lib/xarray.c
create mode 100644 tools/include/asm-generic/bitops/non-atomic.h
create mode 100644 tools/testing/radix-tree/bitmap.c
create mode 100644 tools/testing/radix-tree/linux/kconfig.h
create mode 100644 tools/testing/radix-tree/linux/lockdep.h
create mode 100644 tools/testing/radix-tree/linux/xarray.h
create mode 100644 tools/testing/radix-tree/xarray.c
On Mon, Aug 13, 2018 at 9:14 AM Matthew Wilcox <[email protected]> wrote:
>
> Please consider pulling the XArray patch set.
So this merge window has been horrible, but I was just about to start
looking at it.
And no. I'm not going to pull this.
For some unfathomable reason, you have based it on the libnvdimm tree.
I don't understand at all wjhy you did that.
That libnvdimm tree didn't get merged., because it had complete
garbage in the mm/ code. And yes, that buggy shit was what you based
the radix tree code on.
I seriously have no idea why you have based it on some unstable random
tree in the first place.
But basing it on something that I independently refused to pull
because of obvious bugs from just a quick scan - that completely
invalidates this pull request.
Why?
I guess it makes this merge window easier, since now I don't even have
to look at the code, but it annoys the hell out of me when things like
that happen.
There wasn't even a mention in the pull request about how this was all
based on some libnvdimm code that hadn't been merged yet.
But you must have known that, since you must have explicitly done the
pull request not against my tree, but against the bogus base branch.
And since I won't be merging this, I clearly won't be merging your
other pull request that depended on this either.
Why the f*ck were these features so interlinked to begin with?
Linus
On Tue, Aug 21, 2018 at 7:50 PM Matthew Wilcox <[email protected]> wrote:
>
> So, should I have based just on your tree and sent you a description of
> what a resolved conflict should look like?
Absolutely.
Or preferably not rebasing at all, just starting from a good solid
base for new development in the first place.
Sometimes you start from the wrong point, and decide that you really
need to rebase, but then you should rebase to a more stable point, not
on top of some random independent development.
Rebasing can be a really good tool to clean up development that was
haphazard - maybe as you go along you notice that something you did
earlier turned out to be counter-productive, so you rebase and clean
up your history that has not been made public yet.
But when you send me a big new feature, the absolutely *last* thing I
want to ever see is to see it based on some random unstable base.
And rebasing to avoid merge conflicts is *always* the wrong thing to
do, unless the reason you're rebasing is "hey, I wrote this feature
ages ago, I need to really refresh it to a more modern and stable
kernel, so I'll rebase it onto the current last release instead, so
that I have a good starting point".
And even then the basic reason is not so much that there were
conflicts, but that you just want your series to make more sense on
its own, and not have one horribly complex merge that is just due to
the fact that it was based on something ancient.
The absolute last thing I want to see during the merge window is
multiple independent features that have been tied together just
because they are rebased on top of each other.
Because that means - as in this case - that if one branch has
problems, it now affects all of them.
Merge conflicts aren't bad. In 99% of all cases, the conflict is
trivial to solve. And the cost of trying to deal with them with
rebasing is much much higher.
Linus
On Tue, Aug 21, 2018 at 07:09:31PM -0700, Linus Torvalds wrote:
> On Mon, Aug 13, 2018 at 9:14 AM Matthew Wilcox <[email protected]> wrote:
> >
> > Please consider pulling the XArray patch set.
>
> So this merge window has been horrible, but I was just about to start
> looking at it.
>
> And no. I'm not going to pull this.
>
> For some unfathomable reason, you have based it on the libnvdimm tree.
> I don't understand at all wjhy you did that.
I said in the pull request ...
There are two conflicts I wanted to flag; the first is against the
linux-nvdimm tree. I rebased on top of one of the branches that went
into that tree, so if you pull my tree before linux-nvdimm, you'll get
fifteen commits I've had no involvement with.
Dan asked me to do that so that his commit (which I had no involvement
with) would be easier to backport. At the time I thought this was a
reasonable request; I know this API change is disruptive and I wanted
to accommodate that. I didn't know his patch was "complete garbage";
I didn't review it.
So, should I have based just on your tree and sent you a description of
what a resolved conflict should look like?
> And since I won't be merging this, I clearly won't be merging your
> other pull request that depended on this either.
I can yank most of the patches (all but the last two, iirc) out of the
IDA patchset and submit those as a separate pull request. Would that
be acceptable? I'm really struggling to juggle all the pieces here to
get them merged.
On Mon, 13 Aug 2018, Matthew Wilcox wrote:
> Please consider pulling the XArray patch set. The XArray provides an
> improved interface to the radix tree data structure, providing locking
> as part of the API, specifying GFP flags at allocation time, eliminating
> preloading, less re-walking the tree, more efficient iterations and not
> exposing RCU-protected pointers to its users.
Is this going in this cycle? I have a bunch of stuff on top of this to
enable slab object migration.
On Wed, Aug 22, 2018 at 10:40 AM Christopher Lameter <[email protected]> wrote:
>
> Is this going in this cycle? I have a bunch of stuff on top of this to
> enable slab object migration.
No.
It was based on a buggy branch that isn't getting pulled, so when I
started looking at it, the pull request was rejected before I got much
further.
Linus
On Wed, Aug 22, 2018 at 10:43 AM Linus Torvalds
<[email protected]> wrote:
>
> On Wed, Aug 22, 2018 at 10:40 AM Christopher Lameter <[email protected]> wrote:
> >
> > Is this going in this cycle? I have a bunch of stuff on top of this to
> > enable slab object migration.
>
> No.
>
> It was based on a buggy branch that isn't getting pulled
To be clear, I don't think the problem you identified can be triggered
in practice. We are under the equivalent of the page lock for dax in
that path, and if ->mapping is NULL we would bail before finding that
the mapping-size helper returns zero.
> so when I
> started looking at it, the pull request was rejected before I got much
> further.
For the record I think skipping the entirety of the libnvdimm pull
request for this cycle due to that misuse of ilog2() is overkill, but
it's not my kernel.
Andrew, I think this means we need to lean on you to merge
dax-memory-failure and Xarray for 4.20 rather than try to coordinate
our own git branches for these specific topics.
At a minimum for 4.19 I think we should disable MADV_HWPOISON for dax
mappings this cycle to at least close that trivial method to crash the
kernel when using dax.
Dave, I recommend dropping dax-memory-failure and sending the other
libnvdimm topics for 4.19 that have been soaking in -next.
On Tue, Aug 21, 2018 at 08:00:18PM -0700, Linus Torvalds wrote:
> On Tue, Aug 21, 2018 at 7:50 PM Matthew Wilcox <[email protected]> wrote:
> > So, should I have based just on your tree and sent you a description of
> > what a resolved conflict should look like?
>
> Absolutely.
>
> Or preferably not rebasing at all, just starting from a good solid
> base for new development in the first place.
Ah, I remember now, it was more complex than a textual conflict.
Dan added an entirely new function here:
http://git.infradead.org/users/willy/linux-dax.git/commitdiff/c2a7d2a115525d3501d38e23d24875a79a07e15e
which needed to be converted to XArray. So I should have pulled in his
branch as a merge somewhere close to the end of my series, then done a
fresh patch on top of that to convert it?
It would have been pretty ugly because he modified a function I deleted.
I might try it out just to show how bad it would have been.
On Wed, Aug 22, 2018 at 11:23 AM Matthew Wilcox <[email protected]> wrote:
>
> Dan added an entirely new function here:
>
> http://git.infradead.org/users/willy/linux-dax.git/commitdiff/c2a7d2a115525d3501d38e23d24875a79a07e15e
>
> which needed to be converted to XArray. So I should have pulled in his
> branch as a merge somewhere close to the end of my series, then done a
> fresh patch on top of that to convert it?
No, it doesn't matter if you rebase on top of a broken branch, or you
merge it in. Either way, it's broken and I can't merge your end
result.
You should simply NOT CARE about other branches. Particularly not
other branches that might not even get merged in the first place!
You should care about *YOUR* work. You should make sure *your* work
is rock solid, and that it is based on a rock solid base. Not some
random shifting quick-sand of somebody elses development branch.
Sure, then linux-next will give a merge conflict, but at that point
YOU DO NOT REBASE OR MERGE. You tell linux-next how to merge it, and
mention it to me in the pull request.
Because at that point, I have the *choice* of merging just one of the
branches or both. Or I can merge them in either order, and test them
independently?
See how that is fundamentally different from you tying the two
branches together and handing me a fait accompli?
Yes, yes, sometimes you have to tie branches together because one
branch fundamentally depends on the features the other branch offers.
But that should be avoided like a plague if at all possible.
And it damn well isn't an issue for something like xarray, which has a
life entirely independently of libnvdimm (and vice versa), and the
conflict was just random happenstance, not some "my changes
fundamentally rely on the new features you provide".
Linus
Hi Linus,
On Tue, 21 Aug 2018 19:09:31 -0700 Linus Torvalds <[email protected]> wrote:
>
> For some unfathomable reason, you have based it on the libnvdimm tree.
> I don't understand at all wjhy you did that.
That was partly my fault for giving not very good advice when the
(quite complex) merge conflict turned up in the linux-next tree. I
will try to give better advice in the future.
--
Cheers,
Stephen Rothwell
On 08/22/2018 07:40 PM, Christopher Lameter wrote:
> On Mon, 13 Aug 2018, Matthew Wilcox wrote:
>
>> Please consider pulling the XArray patch set. The XArray provides an
>> improved interface to the radix tree data structure, providing locking
>> as part of the API, specifying GFP flags at allocation time, eliminating
>> preloading, less re-walking the tree, more efficient iterations and not
>> exposing RCU-protected pointers to its users.
>
> Is this going in this cycle? I have a bunch of stuff on top of this to
> enable slab object migration.
I think you can just post those for review and say that they apply on
top of xarray git? Maybe also with your own git URL with those applied
for easier access? I'm curious but also sceptical that something so
major would get picked up to mmotm immediately :)
On Fri, 24 Aug 2018, Vlastimil Babka wrote:
>
> I think you can just post those for review and say that they apply on
> top of xarray git? Maybe also with your own git URL with those applied
> for easier access? I'm curious but also sceptical that something so
> major would get picked up to mmotm immediately :)
>
I posted it awhile ago and was waiting for something definite to diff
against so that testing is simple.
The last release is at
https://www.spinics.net/lists/linux-mm/msg142496.html