Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: [PATCH v2 1/2] drm/ttm: Only allocate huge pages with new flag
 TTM_PAGE_FLAG_TRANSHUGE
To:     =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>,
        Ilia Mirkin <imirkin@alum.mit.edu>
Cc:     amd-gfx mailing list <amd-gfx@lists.freedesktop.org>,
        dri-devel <dri-devel@lists.freedesktop.org>,
        LKML <linux-kernel@vger.kernel.org>
References: <20180426150618.13470-1-michel@daenzer.net>
 <20180427130811.7642-1-michel@daenzer.net>
 <CAKb7Uvj6FFRK6dtvR7xpfZDzs0o3QKHq7ESueBBmr_VnQt6XDQ@mail.gmail.com>
 <e8166443-14f0-1eb3-a6a2-d425792b0643@daenzer.net>
 <d5f9459e-801c-e592-d06d-c9052ef3ad4d@amd.com>
From:   =?UTF-8?Q?Michel_D=c3=a4nzer?= <michel@daenzer.net>
Message-ID: <e5256136-6da0-cc5e-9490-5a70c89dfd83@daenzer.net>
Date:   Mon, 30 Apr 2018 18:33:24 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
 Thunderbird/52.7.0
MIME-Version: 1.0
In-Reply-To: <d5f9459e-801c-e592-d06d-c9052ef3ad4d@amd.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-CA
Content-Transfer-Encoding: 8bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On 2018-04-29 09:02 AM, Christian König wrote:
> Am 29.04.2018 um 01:02 schrieb Michel Dänzer:
>> On 2018-04-28 06:30 PM, Ilia Mirkin wrote:
>>> On Fri, Apr 27, 2018 at 9:08 AM, Michel Dänzer <michel@daenzer.net>
>>> wrote:
>>>> From: Michel Dänzer <michel.daenzer@amd.com>
>>>>
>>>> Previously, TTM would always (with CONFIG_TRANSPARENT_HUGEPAGE enabled)
>>>> try to allocate huge pages. However, not all drivers can take advantage
>>>> of huge pages, but they would incur the overhead for allocating and
>>>> freeing them anyway.
>>>>
>>>> Now, drivers which can take advantage of huge pages need to set the new
>>>> flag TTM_PAGE_FLAG_TRANSHUGE to get them. Drivers not setting this flag
>>>> no longer incur any overhead for allocating or freeing huge pages.
>>>>
>>>> v2:
>>>> * Also guard swapping of consecutive pages in ttm_get_pages
>>>> * Reword commit log, hopefully clearer now
>>>>
>>>> Cc: stable@vger.kernel.org
>>>> Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
>>> Both I and lots of other people, based on reports, are still seeing
>>> plenty of issues with this as late as 4.16.4.
>> "lots of other people", "plenty of issues" sounds a bit exaggerated from
>> what I've seen. FWIW, while I did see the original messages myself, I
>> haven't seen any since Christian's original fix (see below), neither
>> with amdgpu nor radeon, even before this patch you followed up to.
>>
>>
>>> Admittedly I'm on nouveau, but others have reported issues with
>>> radeon/amdgpu as well. It's been going on since the feature was merged
>>> in v4.15, with what seems like little investigation from the authors
>>> introducing the feature.
>> That's not a fair assessment. See
>> https://bugs.freedesktop.org/show_bug.cgi?id=104082#c40 and following
>> comments.
>>
>> Christian fixed the original issue in
>> d0bc0c2a31c95002d37c3cc511ffdcab851b3256 "swiotlb: suppress warning when
>> __GFP_NOWARN is set". Christian did his best to try and get the fix in
>> before 4.15 final, but for reasons beyond his control, it was delayed
>> until 4.16-rc1 and then backported to 4.15.5.
>>
>> Unfortunately, there was an swiotlb regression (not directly related to
>> Christian's work) shortly after this fix, also in 4.16-rc1, which is now
>> fixed in 4.17-rc1 and will be backported to 4.16.y.
> 
> And that's exactly the reason why I intentionally kept this enabled for
> all users of the TTM DMA page pool and not put it behind a flag.
> 
> This change has surfaced quite a number of bugs in the swiotlb code
> which could have caused issues before. It's just that those code path
> where never exercised massively before.
> 
> Additional to that using huge pages is beneficial for the MM and CPU TLB
> (not implemented yet) even when the GPU driver can't make much use of it.

Do I understand correctly that you're against this patch?

AFAIU the only benefit of huge pages with a driver which doesn't take
advantage of them directly is "for the MM". Can you describe a bit more
what that benefit is exactly? Is it expected to outweigh the cost of
allocating / freeing huge pages?


>> It looks like there's at least one more bug left, but it's not clear yet
>> when that was introduced, whether it's directly related to Christian's
>> work, or indeed what the impact is. Let's not get ahead of ourselves.
> 
> Well my patches surfaced the problems, but the underlying issues where
> present even before those changes and I'm very well involved in fixing
> the underlying issues.
> 
> I even considered to just revert the huge page path for the DMA pool
> allocator, but it's just that the TTM patches seem to work exactly as
> they are intended. So that doesn't feel like doing the right thing here.

I agree. Has anyone reported this to the DMA/SWIOTLB developers?


-- 
Earthling Michel Dänzer               |               http://www.amd.com
Libre software enthusiast             |             Mesa and X developer