Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932308AbcDSRHd (ORCPT ); Tue, 19 Apr 2016 13:07:33 -0400 Received: from mail-vk0-f42.google.com ([209.85.213.42]:36377 "EHLO mail-vk0-f42.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932125AbcDSRHb (ORCPT ); Tue, 19 Apr 2016 13:07:31 -0400 MIME-Version: 1.0 In-Reply-To: <20160419165024.GB24312@redhat.com> References: <1460766240-84565-1-git-send-email-kirill.shutemov@linux.intel.com> <571565F0.9070203@linaro.org> <20160419165024.GB24312@redhat.com> Date: Tue, 19 Apr 2016 10:07:29 -0700 Message-ID: Subject: Re: [PATCHv7 00/29] THP-enabled tmpfs/shmem using compound pages From: Andres Lagar-Cavilla To: Andrea Arcangeli Cc: "Shi, Yang" , "Kirill A. Shutemov" , Hugh Dickins , Andrew Morton , Dave Hansen , Vlastimil Babka , Christoph Lameter , Naoya Horiguchi , Jerome Marchand , Sasha Levin , Ning Qu , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3421 Lines: 84 Andrea, we provide the, ahem, adjustments to transparent_hugepage_adjust. Rest assured we aggressively use mmu notifiers with no further changes required. As in: zero changes have been required in the lifetime (years) of kvm+huge tmpfs at Google, other than mod'ing transparent_hugepage_adjust. As noted by Paolo, the additions to transparent_hugepage_adjust could be lifted outside of kvm (into shmem.c? maybe) for any consumer of huge tmpfs with mmu notifiers. Andres On Tue, Apr 19, 2016 at 9:50 AM, Andrea Arcangeli wrote: > Hello, > > On Mon, Apr 18, 2016 at 03:55:44PM -0700, Shi, Yang wrote: >> Hi Kirill, >> >> Finally, I got some time to look into and try yours and Hugh's patches, >> got two problems. > > One thing that come to mind to test is this: qemu with -machine > accel=kvm -mem-path=/dev/shm/,share=on . > > The THP Compound approach in tmpfs may just happen to work already > with KVM (or at worst it'd require minor adjustments) because it uses > the exact same model KVM is already aware about from THP in anonymous > memory, example from arch/x86/kvm/mmu.c: > > static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu, > gfn_t *gfnp, kvm_pfn_t *pfnp, > int *levelp) > { > kvm_pfn_t pfn = *pfnp; > gfn_t gfn = *gfnp; > int level = *levelp; > > /* > * Check if it's a transparent hugepage. If this would be an > * hugetlbfs page, level wouldn't be set to > * PT_PAGE_TABLE_LEVEL and there would be no adjustment done > * here. > */ > if (!is_error_noslot_pfn(pfn) && !kvm_is_reserved_pfn(pfn) && > level == PT_PAGE_TABLE_LEVEL && > PageTransCompound(pfn_to_page(pfn)) && > !mmu_gfn_lpage_is_disallowed(vcpu, gfn, PT_DIRECTORY_LEVEL)) { > > Not using two different models between THP in tmpfs and THP in anon is > essential not just to significantly reduce the size of the kernel > code, but also because THP knowledge can't be self contained in the > mm/shmem.c file. Having to support two different models would > complicate things for secondary MMU drivers (i.e. mmu notifer users) > like KVM who also need to create huge mapping in the shadow pagetable > layer in arch/x86/kvm if the primary MMU allows for it. > >> x86-64 and ARM64 with yours and Hugh's patches (linux-next tree), I got >> the program execution time reduced by ~12% on x86-64, it looks very >> impressive. > > Agreed, both patchset are impressive works and achieving amazing > results! > > My view is that in terms of long-lived computation from userland point > of view, both models are malleable enough and could achieve everything > we need in the end, but as far as the overall kernel efficiency is > concerned the compound model will always retain a slight advantage in > performance by leveraging a native THP compound refcounting that > requires just one atomic_inc/dec per THP mapcount instead of 512 of > them. Other advantages of the compound model is that it's half in code > size despite already including khugepaged (i.e. the same > split_huge_page works for both tmpfs and anon) and like said above it > won't introduce much complications for drivers like KVM as the model > didn't change. > > Thanks, > Andrea -- Andres Lagar-Cavilla | Google Kernel Team | andreslc@google.com