Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Tue, 1 Sep 2020 10:51:55 -0700 (PDT)
From:   Hugh Dickins <hughd@google.com>
To:     "Maoming (maoming, Cloud Infrastructure Service Product Dept.)" 
        <maoming.maoming@huawei.com>
cc:     Hugh Dickins <hughd@google.com>,
        Matthew Wilcox <willy@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "cohuck@redhat.com" <cohuck@redhat.com>,
        "Zhoujian (jay)" <jianjay.zhou@huawei.com>,
        "Huangweidong (C)" <weidong.huang@huawei.com>,
        "peterx@redhat.com" <peterx@redhat.com>,
        "aarcange@redhat.com" <aarcange@redhat.com>,
        wangyunjian <wangyunjian@huawei.com>
Subject: =?UTF-8?Q?Re=3A_=E7=AD=94=E5=A4=8D=3A_=5BPATCH_V3=5D_vfio_dma=5Fmap=2Funmap=3A_optimized_for_hugetlbfs_pages?=
In-Reply-To: <8B561EC9A4D13649A62CF60D3A8E8CB28C2DC466@dggeml524-mbx.china.huawei.com>
Message-ID: <alpine.LSU.2.11.2009011040560.2984@eggly.anvils>
References: <20200828092649.853-1-maoming.maoming@huawei.com> <alpine.LSU.2.11.2008302332330.2382@eggly.anvils> <8B561EC9A4D13649A62CF60D3A8E8CB28C2DC466@dggeml524-mbx.china.huawei.com>
User-Agent: Alpine 2.11 (LSU 23 2013-08-11)
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Tue, 1 Sep 2020, Maoming (maoming, Cloud Infrastructure Service Product Dept.) wrote:
> > 
> > > In the original process of dma_map/unmap pages for VFIO-devices, to
> > > make sure the pages are contiguous, we have to check them one by one.
> > > As a result, dma_map/unmap could spend a long time.
> > > Using the hugetlb pages, we can avoid this problem.
> > > All pages in hugetlb pages are contiguous.And the hugetlb page should
> > > not be split.So we can delete the for loops.
> > 
> > I know nothing about VFIO, but I'm surprised that you're paying such attention
> > to PageHuge hugetlbfs pages, rather than to PageCompound
> > pages: which would also include Transparent Huge Pages, of the traditional
> > anonymous kind, or the huge tmpfs kind, or the more general (not necessarily
> > pmd-sized) kind that Matthew Wilcox is currently working on.
> > 
> > It's true that hugetlbfs is peculiar enough that whatever you write for it may
> > need some tweaks to cover the THP case too, or vice versa; but wouldn't your
> > patch be a lot better for covering all cases?
> > 
> > You mention above that "the hugetlb page should not be split":
> > perhaps you have been worried that a THP could be split beneath you?
> > That used to be a possibility some years ago, but nowadays a THP cannot be
> > split while anyone is pinning it with an extra reference.
> > 
> > Hugh
> > 
> 
> 
> Thanks for your suggestions.
> You mention that a THP cannot be split while anyone is pinning it.
> Do you mean the check of can_split_huge_page()(in split_huge_page_to_list())?

Partly, yes: but that's just a racy check to avoid doing wasted work in
common cases; the more important check comes later in page_ref_freeze(),
which either fails, or prevents anyone else taking a new reference to
the THP (head or tails) while all the work on splitting is being done.

> When we want to pin pages, vfio_pin_pages_remote() always gets a normal page first.
> In this case, a THP cannot be split because of the increased reference.
> Is this right?

Yes.

> 
> Maybe I can optimize for hugetlb pages as a first step,
> and then cover all compound pages.

Okay.  That's definitely a good way to start out - but you may find
that the second step leads you to rewrite every line you wrote before!

Hugh