576ed913 "block: use bio_add_page in bio_iov_iter_get_pages", applied
late in the 4.19 cycle appears to introduce a regression causing a
huge page leak in a complicated set of circumstances I haven't fully
identified yet.
On a POWER8 machine with a kernel after the commit above, when I run a
KVM guest with RAM in hugetlbfs pages (and certain options, see below), a
handful of the hugepages used for RAM are not released after qemu and
the guest quit. Usually 2 or 3 16MiB pages are leaked, though I've
seen anything from 0-8 occasionally.
There are a bunch of conditions on when it occurs, only some of which
I've pinned down:
* It happens on a POWER8 8247-22L, but not a very similar 8247-21L,
and I haven't been able to work out why, yet.
* It only happens with certain combination of qemu block and caching
options for the guest's root fs. Specifically it appears to happen
when the file used for the guest's root disk image is opened with
O_DIRECT.
* It depends somewhat on guest activity.
- It doesn't occur if the guest is only booted to firmware
- Booting only to initramfs without mounting the "real" root fs
doesn't seem to trigger the problem
- It appears to happen reliably with RHEL6 and RHEL7 guests, but
only sometimes with RHEL8 guests, again, I don't know why at
this stage
I pinned it down to this (host kernel) patch by bisection, and I've
double checked afterwards to confirm it really is this commit, not a
mistake during the bisection.
I've tried a bunch of instrumentation, but it hasn't been very
illuminating so far:
* The leaked pages have non-zero count and are left in the
hugepage_activelist
* The leaked pages *don't* appear to be blocking release of the KVM
VM or the qemu process owning it
* The leaked pages *do* appear to be blocking release of the
associated address_space and (anonymous) inode, though I'm not 100%
certain about this.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
Just curious: What exact trees do you see this with? This area
changed a lot with the multipage bvec support, and subsequent fixes.
So I'd be really curious if it can be reproduced with Jens' latest
block for-5.2 tree (which should be in latest linux-next).
On Thu, Apr 25, 2019 at 08:19:58AM +0200, Christoph Hellwig wrote:
> Just curious: What exact trees do you see this with? This area
> changed a lot with the multipage bvec support, and subsequent fixes.
So, I tried it with 576ed913 itself and with 576ed913^ to verify that
it didn't happen there. The problem also occurred with Linus' tree as
of when I started bisecting, which appears to have been 444fe991.
Actually, come to that, here's the whole bisect log in case it's
helpful:
# git bisect log
git bisect start
# good: [bebc6082da0a9f5d47a1ea2edc099bf671058bd4] Linux 4.14
git bisect good bebc6082da0a9f5d47a1ea2edc099bf671058bd4
# bad: [444fe991353987c1c9bc5ab1f903d01f1b4ad415] Merge tag 'riscv-for-linus-5.1-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/palmer/riscv-linux
git bisect bad 444fe991353987c1c9bc5ab1f903d01f1b4ad415
# good: [399c4129eba6145924ab90363352b7bdcd554751] Merge tag 'pxa-for-4.19-dma_slave_map' of https://github.com/rjarzmik/linux
git bisect good 399c4129eba6145924ab90363352b7bdcd554751
# bad: [73b6f96cbc0162787bcbdac5f538167084c8d605] Merge branch 'drm-fixes-4.20' of git://people.freedesktop.org/~agd5f/linux into drm-fixes
git bisect bad 73b6f96cbc0162787bcbdac5f538167084c8d605
# good: [85a585918fb4122ad26b6febaec5c3c90bf2535c] Merge tag 'loadpin-security-next' of https://git.kernel.org/pub/scm/linux/kernel/git/kees/linux into next-loadpin
git bisect good 85a585918fb4122ad26b6febaec5c3c90bf2535c
# bad: [3acbd2de6bc3af215c6ed7732dfc097d1e238503] Merge tag 'sound-4.20-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound
git bisect bad 3acbd2de6bc3af215c6ed7732dfc097d1e238503
# good: [8f18da47211554f1ef674fef627c05f23b75a8e0] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/klassert/ipsec-next
git bisect good 8f18da47211554f1ef674fef627c05f23b75a8e0
# bad: [0d1b82cd8ac2e8856ae9045c97782ac1c359929c] Merge branch 'ras-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad 0d1b82cd8ac2e8856ae9045c97782ac1c359929c
# bad: [1650ac53066577a5e83fe3e9d992c9311597ff8c] Merge tag 'mmc-v4.20' of git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc
git bisect bad 1650ac53066577a5e83fe3e9d992c9311597ff8c
# bad: [6ab9e09238fdfd742fe23b81e2d385a1cab49d9b] Merge tag 'for-4.20/block-20181021' of git://git.kernel.dk/linux-block
git bisect bad 6ab9e09238fdfd742fe23b81e2d385a1cab49d9b
# good: [528985117126f11beea339cf39120ee99da04cd2] Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
git bisect good 528985117126f11beea339cf39120ee99da04cd2
# bad: [2cf99bbd106f89fc72f778e8ad9d5538f1ef939b] lightnvm: pblk: add helpers for chunk addresses
git bisect bad 2cf99bbd106f89fc72f778e8ad9d5538f1ef939b
# bad: [33b14f67a4e1eabd219fd6543da8f15ed86b641c] nvme: register ns_id attributes as default sysfs groups
git bisect bad 33b14f67a4e1eabd219fd6543da8f15ed86b641c
# bad: [27ca1d4ed04ea29dc77b47190a3cc82697023e76] block: move req_gap_back_merge to blk.h
git bisect bad 27ca1d4ed04ea29dc77b47190a3cc82697023e76
# bad: [07b05bcc3213ac9f8c28c9d835b4bf3d5798cc60] blkcg: convert blkg_lookup_create to find closest blkg
git bisect bad 07b05bcc3213ac9f8c28c9d835b4bf3d5798cc60
# good: [cbeb869a3d1110450186b738199963c5e68c2a71] block, bfq: correctly charge and reset entity service in all cases
git bisect good cbeb869a3d1110450186b738199963c5e68c2a71
# bad: [576ed9135489c723fb39b97c4e2c73428d06dd78] block: use bio_add_page in bio_iov_iter_get_pages
git bisect bad 576ed9135489c723fb39b97c4e2c73428d06dd78
# good: [c8765de0adfcaaf4ffb2d951e07444f00ffa9453] blok, bfq: do not plug I/O if all queues are weight-raised
git bisect good c8765de0adfcaaf4ffb2d951e07444f00ffa9453
# first bad commit: [576ed9135489c723fb39b97c4e2c73428d06dd78] block: use bio_add_page in bio_iov_iter_get_pages
The problem also occurred with the RHEL8 downstream kernel tree.
That's based on 4.18, but has 576ed913 backported.
> So I'd be really curious if it can be reproduced with Jens' latest
> block for-5.2 tree (which should be in latest linux-next).
I'll see if I can try that when I next get access to the machine.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
On Fri, Apr 26, 2019 at 11:05:17AM +1000, David Gibson wrote:
> On Thu, Apr 25, 2019 at 08:19:58AM +0200, Christoph Hellwig wrote:
> > Just curious: What exact trees do you see this with? This area
> > changed a lot with the multipage bvec support, and subsequent fixes.
>
> So, I tried it with 576ed913 itself and with 576ed913^ to verify that
> it didn't happen there. The problem also occurred with Linus' tree as
> of when I started bisecting, which appears to have been 444fe991.
> Actually, come to that, here's the whole bisect log in case it's
> helpful:
>
> # git bisect log
> git bisect start
> # good: [bebc6082da0a9f5d47a1ea2edc099bf671058bd4] Linux 4.14
> git bisect good bebc6082da0a9f5d47a1ea2edc099bf671058bd4
> # bad: [444fe991353987c1c9bc5ab1f903d01f1b4ad415] Merge tag 'riscv-for-linus-5.1-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/palmer/riscv-linux
> git bisect bad 444fe991353987c1c9bc5ab1f903d01f1b4ad415
> # good: [399c4129eba6145924ab90363352b7bdcd554751] Merge tag 'pxa-for-4.19-dma_slave_map' of https://github.com/rjarzmik/linux
> git bisect good 399c4129eba6145924ab90363352b7bdcd554751
> # bad: [73b6f96cbc0162787bcbdac5f538167084c8d605] Merge branch 'drm-fixes-4.20' of git://people.freedesktop.org/~agd5f/linux into drm-fixes
> git bisect bad 73b6f96cbc0162787bcbdac5f538167084c8d605
> # good: [85a585918fb4122ad26b6febaec5c3c90bf2535c] Merge tag 'loadpin-security-next' of https://git.kernel.org/pub/scm/linux/kernel/git/kees/linux into next-loadpin
> git bisect good 85a585918fb4122ad26b6febaec5c3c90bf2535c
> # bad: [3acbd2de6bc3af215c6ed7732dfc097d1e238503] Merge tag 'sound-4.20-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound
> git bisect bad 3acbd2de6bc3af215c6ed7732dfc097d1e238503
> # good: [8f18da47211554f1ef674fef627c05f23b75a8e0] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/klassert/ipsec-next
> git bisect good 8f18da47211554f1ef674fef627c05f23b75a8e0
> # bad: [0d1b82cd8ac2e8856ae9045c97782ac1c359929c] Merge branch 'ras-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
> git bisect bad 0d1b82cd8ac2e8856ae9045c97782ac1c359929c
> # bad: [1650ac53066577a5e83fe3e9d992c9311597ff8c] Merge tag 'mmc-v4.20' of git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc
> git bisect bad 1650ac53066577a5e83fe3e9d992c9311597ff8c
> # bad: [6ab9e09238fdfd742fe23b81e2d385a1cab49d9b] Merge tag 'for-4.20/block-20181021' of git://git.kernel.dk/linux-block
> git bisect bad 6ab9e09238fdfd742fe23b81e2d385a1cab49d9b
> # good: [528985117126f11beea339cf39120ee99da04cd2] Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
> git bisect good 528985117126f11beea339cf39120ee99da04cd2
> # bad: [2cf99bbd106f89fc72f778e8ad9d5538f1ef939b] lightnvm: pblk: add helpers for chunk addresses
> git bisect bad 2cf99bbd106f89fc72f778e8ad9d5538f1ef939b
> # bad: [33b14f67a4e1eabd219fd6543da8f15ed86b641c] nvme: register ns_id attributes as default sysfs groups
> git bisect bad 33b14f67a4e1eabd219fd6543da8f15ed86b641c
> # bad: [27ca1d4ed04ea29dc77b47190a3cc82697023e76] block: move req_gap_back_merge to blk.h
> git bisect bad 27ca1d4ed04ea29dc77b47190a3cc82697023e76
> # bad: [07b05bcc3213ac9f8c28c9d835b4bf3d5798cc60] blkcg: convert blkg_lookup_create to find closest blkg
> git bisect bad 07b05bcc3213ac9f8c28c9d835b4bf3d5798cc60
> # good: [cbeb869a3d1110450186b738199963c5e68c2a71] block, bfq: correctly charge and reset entity service in all cases
> git bisect good cbeb869a3d1110450186b738199963c5e68c2a71
> # bad: [576ed9135489c723fb39b97c4e2c73428d06dd78] block: use bio_add_page in bio_iov_iter_get_pages
> git bisect bad 576ed9135489c723fb39b97c4e2c73428d06dd78
> # good: [c8765de0adfcaaf4ffb2d951e07444f00ffa9453] blok, bfq: do not plug I/O if all queues are weight-raised
> git bisect good c8765de0adfcaaf4ffb2d951e07444f00ffa9453
> # first bad commit: [576ed9135489c723fb39b97c4e2c73428d06dd78] block: use bio_add_page in bio_iov_iter_get_pages
>
> The problem also occurred with the RHEL8 downstream kernel tree.
> That's based on 4.18, but has 576ed913 backported.
>
> > So I'd be really curious if it can be reproduced with Jens' latest
> > block for-5.2 tree (which should be in latest linux-next).
>
> I'll see if I can try that when I next get access to the machine.
Ok, I've now had a chance to test the next-20190423 tree.
I can still reproduce the problem: in fact it is substantially worse,
and somewhat more consistent.
Previously I usually lost 2-3 hugepages per run, though I'd
occasionally seen other values between 0 and 8. With the next tree, I
lost 46 hugepages on most runs, though I also saw 45 and 48
occasionally.
Hope this is illuminating.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
On Fri, Apr 26, 2019 at 12:41 PM David Gibson
<[email protected]> wrote:
>
> On Fri, Apr 26, 2019 at 11:05:17AM +1000, David Gibson wrote:
> > On Thu, Apr 25, 2019 at 08:19:58AM +0200, Christoph Hellwig wrote:
> > > Just curious: What exact trees do you see this with? This area
> > > changed a lot with the multipage bvec support, and subsequent fixes.
> >
> > So, I tried it with 576ed913 itself and with 576ed913^ to verify that
> > it didn't happen there. The problem also occurred with Linus' tree as
> > of when I started bisecting, which appears to have been 444fe991.
> > Actually, come to that, here's the whole bisect log in case it's
> > helpful:
> >
> > # git bisect log
> > git bisect start
> > # good: [bebc6082da0a9f5d47a1ea2edc099bf671058bd4] Linux 4.14
> > git bisect good bebc6082da0a9f5d47a1ea2edc099bf671058bd4
> > # bad: [444fe991353987c1c9bc5ab1f903d01f1b4ad415] Merge tag 'riscv-for-linus-5.1-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/palmer/riscv-linux
> > git bisect bad 444fe991353987c1c9bc5ab1f903d01f1b4ad415
> > # good: [399c4129eba6145924ab90363352b7bdcd554751] Merge tag 'pxa-for-4.19-dma_slave_map' of https://github.com/rjarzmik/linux
> > git bisect good 399c4129eba6145924ab90363352b7bdcd554751
> > # bad: [73b6f96cbc0162787bcbdac5f538167084c8d605] Merge branch 'drm-fixes-4.20' of git://people.freedesktop.org/~agd5f/linux into drm-fixes
> > git bisect bad 73b6f96cbc0162787bcbdac5f538167084c8d605
> > # good: [85a585918fb4122ad26b6febaec5c3c90bf2535c] Merge tag 'loadpin-security-next' of https://git.kernel.org/pub/scm/linux/kernel/git/kees/linux into next-loadpin
> > git bisect good 85a585918fb4122ad26b6febaec5c3c90bf2535c
> > # bad: [3acbd2de6bc3af215c6ed7732dfc097d1e238503] Merge tag 'sound-4.20-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound
> > git bisect bad 3acbd2de6bc3af215c6ed7732dfc097d1e238503
> > # good: [8f18da47211554f1ef674fef627c05f23b75a8e0] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/klassert/ipsec-next
> > git bisect good 8f18da47211554f1ef674fef627c05f23b75a8e0
> > # bad: [0d1b82cd8ac2e8856ae9045c97782ac1c359929c] Merge branch 'ras-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
> > git bisect bad 0d1b82cd8ac2e8856ae9045c97782ac1c359929c
> > # bad: [1650ac53066577a5e83fe3e9d992c9311597ff8c] Merge tag 'mmc-v4.20' of git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc
> > git bisect bad 1650ac53066577a5e83fe3e9d992c9311597ff8c
> > # bad: [6ab9e09238fdfd742fe23b81e2d385a1cab49d9b] Merge tag 'for-4.20/block-20181021' of git://git.kernel.dk/linux-block
> > git bisect bad 6ab9e09238fdfd742fe23b81e2d385a1cab49d9b
> > # good: [528985117126f11beea339cf39120ee99da04cd2] Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
> > git bisect good 528985117126f11beea339cf39120ee99da04cd2
> > # bad: [2cf99bbd106f89fc72f778e8ad9d5538f1ef939b] lightnvm: pblk: add helpers for chunk addresses
> > git bisect bad 2cf99bbd106f89fc72f778e8ad9d5538f1ef939b
> > # bad: [33b14f67a4e1eabd219fd6543da8f15ed86b641c] nvme: register ns_id attributes as default sysfs groups
> > git bisect bad 33b14f67a4e1eabd219fd6543da8f15ed86b641c
> > # bad: [27ca1d4ed04ea29dc77b47190a3cc82697023e76] block: move req_gap_back_merge to blk.h
> > git bisect bad 27ca1d4ed04ea29dc77b47190a3cc82697023e76
> > # bad: [07b05bcc3213ac9f8c28c9d835b4bf3d5798cc60] blkcg: convert blkg_lookup_create to find closest blkg
> > git bisect bad 07b05bcc3213ac9f8c28c9d835b4bf3d5798cc60
> > # good: [cbeb869a3d1110450186b738199963c5e68c2a71] block, bfq: correctly charge and reset entity service in all cases
> > git bisect good cbeb869a3d1110450186b738199963c5e68c2a71
> > # bad: [576ed9135489c723fb39b97c4e2c73428d06dd78] block: use bio_add_page in bio_iov_iter_get_pages
> > git bisect bad 576ed9135489c723fb39b97c4e2c73428d06dd78
> > # good: [c8765de0adfcaaf4ffb2d951e07444f00ffa9453] blok, bfq: do not plug I/O if all queues are weight-raised
> > git bisect good c8765de0adfcaaf4ffb2d951e07444f00ffa9453
> > # first bad commit: [576ed9135489c723fb39b97c4e2c73428d06dd78] block: use bio_add_page in bio_iov_iter_get_pages
> >
> > The problem also occurred with the RHEL8 downstream kernel tree.
> > That's based on 4.18, but has 576ed913 backported.
> >
> > > So I'd be really curious if it can be reproduced with Jens' latest
> > > block for-5.2 tree (which should be in latest linux-next).
> >
> > I'll see if I can try that when I next get access to the machine.
>
> Ok, I've now had a chance to test the next-20190423 tree.
>
> I can still reproduce the problem: in fact it is substantially worse,
> and somewhat more consistent.
>
> Previously I usually lost 2-3 hugepages per run, though I'd
> occasionally seen other values between 0 and 8. With the next tree, I
> lost 46 hugepages on most runs, though I also saw 45 and 48
> occasionally.
Please try to apply the following three patches, and I guess you may
see same behavior as linus tree(v5.1-rc, or v4.18). And the last one has
been applied to for-5.2/block which should be the latest next.
https://lore.kernel.org/linux-block/[email protected]/T/#u
https://lore.kernel.org/linux-block/[email protected]/T/#u
https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/commit/?h=for-5.2/block&id=0257c0ed5ea3de3e32cb322852c4c40bc09d1b97
Thanks,
Ming Lei
On Fri, Apr 26, 2019 at 12:41 PM David Gibson
<[email protected]> wrote:
>
> On Fri, Apr 26, 2019 at 11:05:17AM +1000, David Gibson wrote:
> > On Thu, Apr 25, 2019 at 08:19:58AM +0200, Christoph Hellwig wrote:
> > > Just curious: What exact trees do you see this with? This area
> > > changed a lot with the multipage bvec support, and subsequent fixes.
> >
> > So, I tried it with 576ed913 itself and with 576ed913^ to verify that
> > it didn't happen there. The problem also occurred with Linus' tree as
> > of when I started bisecting, which appears to have been 444fe991.
> > Actually, come to that, here's the whole bisect log in case it's
> > helpful:
> >
> > # git bisect log
> > git bisect start
> > # good: [bebc6082da0a9f5d47a1ea2edc099bf671058bd4] Linux 4.14
> > git bisect good bebc6082da0a9f5d47a1ea2edc099bf671058bd4
> > # bad: [444fe991353987c1c9bc5ab1f903d01f1b4ad415] Merge tag 'riscv-for-linus-5.1-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/palmer/riscv-linux
> > git bisect bad 444fe991353987c1c9bc5ab1f903d01f1b4ad415
> > # good: [399c4129eba6145924ab90363352b7bdcd554751] Merge tag 'pxa-for-4.19-dma_slave_map' of https://github.com/rjarzmik/linux
> > git bisect good 399c4129eba6145924ab90363352b7bdcd554751
> > # bad: [73b6f96cbc0162787bcbdac5f538167084c8d605] Merge branch 'drm-fixes-4.20' of git://people.freedesktop.org/~agd5f/linux into drm-fixes
> > git bisect bad 73b6f96cbc0162787bcbdac5f538167084c8d605
> > # good: [85a585918fb4122ad26b6febaec5c3c90bf2535c] Merge tag 'loadpin-security-next' of https://git.kernel.org/pub/scm/linux/kernel/git/kees/linux into next-loadpin
> > git bisect good 85a585918fb4122ad26b6febaec5c3c90bf2535c
> > # bad: [3acbd2de6bc3af215c6ed7732dfc097d1e238503] Merge tag 'sound-4.20-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound
> > git bisect bad 3acbd2de6bc3af215c6ed7732dfc097d1e238503
> > # good: [8f18da47211554f1ef674fef627c05f23b75a8e0] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/klassert/ipsec-next
> > git bisect good 8f18da47211554f1ef674fef627c05f23b75a8e0
> > # bad: [0d1b82cd8ac2e8856ae9045c97782ac1c359929c] Merge branch 'ras-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
> > git bisect bad 0d1b82cd8ac2e8856ae9045c97782ac1c359929c
> > # bad: [1650ac53066577a5e83fe3e9d992c9311597ff8c] Merge tag 'mmc-v4.20' of git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc
> > git bisect bad 1650ac53066577a5e83fe3e9d992c9311597ff8c
> > # bad: [6ab9e09238fdfd742fe23b81e2d385a1cab49d9b] Merge tag 'for-4.20/block-20181021' of git://git.kernel.dk/linux-block
> > git bisect bad 6ab9e09238fdfd742fe23b81e2d385a1cab49d9b
> > # good: [528985117126f11beea339cf39120ee99da04cd2] Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
> > git bisect good 528985117126f11beea339cf39120ee99da04cd2
> > # bad: [2cf99bbd106f89fc72f778e8ad9d5538f1ef939b] lightnvm: pblk: add helpers for chunk addresses
> > git bisect bad 2cf99bbd106f89fc72f778e8ad9d5538f1ef939b
> > # bad: [33b14f67a4e1eabd219fd6543da8f15ed86b641c] nvme: register ns_id attributes as default sysfs groups
> > git bisect bad 33b14f67a4e1eabd219fd6543da8f15ed86b641c
> > # bad: [27ca1d4ed04ea29dc77b47190a3cc82697023e76] block: move req_gap_back_merge to blk.h
> > git bisect bad 27ca1d4ed04ea29dc77b47190a3cc82697023e76
> > # bad: [07b05bcc3213ac9f8c28c9d835b4bf3d5798cc60] blkcg: convert blkg_lookup_create to find closest blkg
> > git bisect bad 07b05bcc3213ac9f8c28c9d835b4bf3d5798cc60
> > # good: [cbeb869a3d1110450186b738199963c5e68c2a71] block, bfq: correctly charge and reset entity service in all cases
> > git bisect good cbeb869a3d1110450186b738199963c5e68c2a71
> > # bad: [576ed9135489c723fb39b97c4e2c73428d06dd78] block: use bio_add_page in bio_iov_iter_get_pages
> > git bisect bad 576ed9135489c723fb39b97c4e2c73428d06dd78
> > # good: [c8765de0adfcaaf4ffb2d951e07444f00ffa9453] blok, bfq: do not plug I/O if all queues are weight-raised
> > git bisect good c8765de0adfcaaf4ffb2d951e07444f00ffa9453
> > # first bad commit: [576ed9135489c723fb39b97c4e2c73428d06dd78] block: use bio_add_page in bio_iov_iter_get_pages
> >
> > The problem also occurred with the RHEL8 downstream kernel tree.
> > That's based on 4.18, but has 576ed913 backported.
> >
> > > So I'd be really curious if it can be reproduced with Jens' latest
> > > block for-5.2 tree (which should be in latest linux-next).
> >
> > I'll see if I can try that when I next get access to the machine.
>
> Ok, I've now had a chance to test the next-20190423 tree.
>
> I can still reproduce the problem: in fact it is substantially worse,
> and somewhat more consistent.
>
> Previously I usually lost 2-3 hugepages per run, though I'd
> occasionally seen other values between 0 and 8. With the next tree, I
> lost 46 hugepages on most runs, though I also saw 45 and 48
> occasionally.
Hi David,
The following two patches should fix the issue, please test it.
https://lore.kernel.org/linux-block/[email protected]/T/#t
Thanks,
Ming Lei