Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Content-Type: text/plain;
        charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\))
Subject: Re: [PATCH v2] iommu/amd: Disable IOMMU on Stoney Ridge systems
From:   Kai-Heng Feng <kai.heng.feng@canonical.com>
In-Reply-To: <C4ADFDF0-8252-412A-8CFE-8E5ACE853B0A@canonical.com>
Date:   Wed, 5 Feb 2020 18:10:11 +0800
Cc:     Christoph Hellwig <hch@infradead.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        Kernel development list <linux-kernel@vger.kernel.org>
Content-Transfer-Encoding: 8BIT
Message-Id: <6B7AE45E-F1E0-4949-B3E6-B78658CD223F@canonical.com>
References: <20191129142154.29658-1-kai.heng.feng@canonical.com>
 <20191202170011.GC30032@infradead.org>
 <974A8EB3-70B6-4A33-B36C-CFF69464493C@canonical.com>
 <20191217095341.GG8689@8bytes.org>
 <6DC0EAB3-89B5-4A16-9A38-D7AD954DDF1C@canonical.com>
 <CY4PR12MB13505BE6EFF95F7C48253120F7520@CY4PR12MB1350.namprd12.prod.outlook.com>
 <84CFD1EE-2DB7-451F-98E4-58C4B0046A81@canonical.com>
 <C4ADFDF0-8252-412A-8CFE-8E5ACE853B0A@canonical.com>
To:     Joerg Roedel <joro@8bytes.org>,
        "Deucher, Alexander" <Alexander.Deucher@amd.com>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

Hi Joerg,

> On Jan 6, 2020, at 16:37, Kai-Heng Feng <kai.heng.feng@canonical.com> wrote:
> 
> 
> 
>> On Dec 20, 2019, at 10:13, Kai-Heng Feng <kai.heng.feng@canonical.com> wrote:
>> 
>> 
>> 
>>> On Dec 20, 2019, at 03:15, Deucher, Alexander <Alexander.Deucher@amd.com> wrote:
>>> 
>>>> -----Original Message-----
>>>> From: Kai-Heng Feng <kai.heng.feng@canonical.com>
>>>> Sent: Wednesday, December 18, 2019 12:45 PM
>>>> To: Joerg Roedel <joro@8bytes.org>
>>>> Cc: Christoph Hellwig <hch@infradead.org>; Deucher, Alexander
>>>> <Alexander.Deucher@amd.com>; iommu@lists.linux-foundation.org; Kernel
>>>> development list <linux-kernel@vger.kernel.org>
>>>> Subject: Re: [PATCH v2] iommu/amd: Disable IOMMU on Stoney Ridge
>>>> systems
>>>> 
>>>> 
>>>> 
>>>>> On Dec 17, 2019, at 17:53, Joerg Roedel <joro@8bytes.org> wrote:
>>>>> 
>>>>> On Fri, Dec 06, 2019 at 01:57:41PM +0800, Kai-Heng Feng wrote:
>>>>>> Hi Joerg,
>>>>>> 
>>>>>>> On Dec 3, 2019, at 01:00, Christoph Hellwig <hch@infradead.org> wrote:
>>>>>>> 
>>>>>>> On Fri, Nov 29, 2019 at 10:21:54PM +0800, Kai-Heng Feng wrote:
>>>>>>>> Serious screen flickering when Stoney Ridge outputs to a 4K monitor.
>>>>>>>> 
>>>>>>>> According to Alex Deucher, IOMMU isn't enabled on Windows, so let's
>>>>>>>> do the same here to avoid screen flickering on 4K monitor.
>>>>>>> 
>>>>>>> Disabling the IOMMU entirely seem pretty severe.  Isn't it enough to
>>>>>>> identity map the GPU device?
>>>>>> 
>>>>>> Ok, there's set_device_exclusion_range() to exclude the device from
>>>> IOMMU.
>>>>>> However I don't know how to generate range_start and range_length,
>>>> which are read from ACPI.
>>>>> 
>>>>> set_device_exclusion_range() is not the solution here. The best is if
>>>>> the GPU device is put into a passthrough domain at boot, in which it
>>>>> will be identity mapped. DMA still goes through the IOMMU in this
>>>>> case, but it only needs to lookup the device-table, page-table walks
>>>>> will not be done anymore.
>>>>> 
>>>>> The best way to implement this is to put it into the
>>>>> amd_iommu_add_device() in drivers/iommu/amd_iommu.c. There is this
>>>>> check:
>>>>> 
>>>>>     if (dev_data->iommu_v2)
>>>>> 		iommu_request_dm_for_dev(dev);
>>>>> 
>>>>> The iommu_request_dm_for_dev() function causes the device to be
>>>>> identity mapped. The check can be extended to also check for a device
>>>>> white-list for devices that need identity mapping.
>>>> 
>>>> My patch looks like this but the original behavior (4K screen flickering) is still
>>>> the same:
>>> 
>>> Does reverting the patch to disable ATS along with this patch help?
>> 
>> Unfortunately it doesn't help.
> 
> Any further suggestion to let me try?

Since using identity mapping with ATS doesn't help,
Is it possible to merge this patch as is?

Kai-Heng

> 
> Kai-Heng
> 
>> 
>> Kai-Heng
>> 
>>> 
>>> Alex
>>> 
>>>> 
>>>> diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c
>>>> index bd25674ee4db..f913a25c9e92 100644
>>>> --- a/drivers/iommu/amd_iommu.c
>>>> +++ b/drivers/iommu/amd_iommu.c
>>>> @@ -42,6 +42,7 @@
>>>> #include <asm/iommu.h>
>>>> #include <asm/gart.h>
>>>> #include <asm/dma.h>
>>>> +#include <asm/pci-direct.h>
>>>> 
>>>> #include "amd_iommu_proto.h"
>>>> #include "amd_iommu_types.h"
>>>> @@ -2159,6 +2160,8 @@ static int amd_iommu_add_device(struct device
>>>> *dev)
>>>>      struct iommu_domain *domain;
>>>>      struct amd_iommu *iommu;
>>>>      int ret, devid;
>>>> +       bool need_identity_mapping = false;
>>>> +       u32 header;
>>>> 
>>>>      if (!check_device(dev) || get_dev_data(dev))
>>>>              return 0;
>>>> @@ -2184,7 +2187,11 @@ static int amd_iommu_add_device(struct device
>>>> *dev)
>>>> 
>>>>      BUG_ON(!dev_data);
>>>> 
>>>> -       if (dev_data->iommu_v2)
>>>> +       header = read_pci_config(0, PCI_BUS_NUM(devid), PCI_SLOT(devid),
>>>> PCI_FUNC(devid));
>>>> +       if ((header & 0xffff) == 0x1002 && (header >> 16) == 0x98e4)
>>>> +               need_identity_mapping = true;
>>>> +
>>>> +       if (dev_data->iommu_v2 || need_identity_mapping)
>>>>              iommu_request_dm_for_dev(dev);
>>>> 
>>>>      /* Domains are initialized for this device - have a look what we ended up
>>>> with */
>>>> 
>>>> 
>>>> $ dmesg | grep -i direct
>>>> [    0.011446] Using GB pages for direct mapping
>>>> [    0.703369] pci 0000:00:01.0: Using iommu direct mapping
>>>> [    0.703830] pci 0000:00:08.0: Using iommu direct mapping
>>>> 
>>>> So the graphics device (pci 0000:00:01.0:) is using direct mapping after the
>>>> change.
>>>> 
>>>> Kai-Heng
>>>> 
>>>>> 
>>>>> HTH,
>>>>> 
>>>>> 	Joerg
>