Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   "Merger, Edgar [AUTOSOL/MAS/AUGS]" <Edgar.Merger@emerson.com>
To:     Huang Rui <ray.huang@amd.com>,
        "Kuehling, Felix" <Felix.Kuehling@amd.com>
CC:     Will Deacon <will@kernel.org>,
        "Deucher, Alexander" <Alexander.Deucher@amd.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Joerg Roedel <jroedel@suse.de>,
        Changfeng Zhu <changfeng.zhu@amd.com>
Subject: RE: [EXTERNAL] Re: [PATCH] PCI: Mark AMD Raven iGPU ATS as broken
Thread-Topic: [EXTERNAL] Re: [PATCH] PCI: Mark AMD Raven iGPU ATS as broken
Thread-Index: AQHWwdw3n5VLcueYHUSP0T6AMdoDgKnWTbwAgAAE0oCAAIPUgIAACwGQ
Date:   Tue, 24 Nov 2020 07:28:42 +0000
Message-ID: <MWHPR10MB13108B04F4765EA6E278660B89FB0@MWHPR10MB1310.namprd10.prod.outlook.com>
References: <20201123134410.10648-1-will@kernel.org>
 <MN2PR12MB4488308D26DB50C18EA3BE0FF7FC0@MN2PR12MB4488.namprd12.prod.outlook.com>
 <20201123223356.GC12069@willie-the-truck>
 <218017ab-defd-c77d-9055-286bf49bee86@amd.com>
 <20201124064301.GA536919@hr-amd>
In-Reply-To: <20201124064301.GA536919@hr-amd>
Accept-Language: de-DE, en-US
Content-Language: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: ea24b056-987e-4250-562e-08d8904a925d
X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Nov 2020 07:28:42.9565
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: eb06985d-06ca-4a17-81da-629ab99f6505
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: ihVWXgbDbRnVG77/qHST27LLIUBQ2C9Noun7d8zhQFw6cg2b4yLgN4yfkBAaai6GKlBKfMv5IwcaPeCRB/rUZSCvVYfoHnbTn5iBYYtLepw=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR1001MB2191
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.312,18.0.737
 definitions=2020-11-24_03:2020-11-24,2020-11-23 signatures=0
X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 mlxlogscore=999
 impostorscore=0 spamscore=0 suspectscore=0 priorityscore=1501
 clxscore=1011 mlxscore=0 lowpriorityscore=0 bulkscore=0 malwarescore=0
 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2009150000 definitions=main-2011240043
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Module Version : PiccasoCpu 10=20
AGESA Version   : PiccasoPI 100A

I did not try to enter the system in any other way (like via ssh) than via =
Desktop.

-----Original Message-----
From: Huang Rui <ray.huang@amd.com>=20
Sent: Dienstag, 24. November 2020 07:43
To: Kuehling, Felix <Felix.Kuehling@amd.com>
Cc: Will Deacon <will@kernel.org>; Deucher, Alexander <Alexander.Deucher@am=
d.com>; linux-kernel@vger.kernel.org; linux-pci@vger.kernel.org; iommu@list=
s.linux-foundation.org; Bjorn Helgaas <bhelgaas@google.com>; Merger, Edgar =
[AUTOSOL/MAS/AUGS] <Edgar.Merger@emerson.com>; Joerg Roedel <jroedel@suse.d=
e>; Changfeng Zhu <changfeng.zhu@amd.com>
Subject: [EXTERNAL] Re: [PATCH] PCI: Mark AMD Raven iGPU ATS as broken

On Tue, Nov 24, 2020 at 06:51:11AM +0800, Kuehling, Felix wrote:
> On 2020-11-23 5:33 p.m., Will Deacon wrote:
> > On Mon, Nov 23, 2020 at 09:04:14PM +0000, Deucher, Alexander wrote:
> >> [AMD Public Use]
> >>
> >>> -----Original Message-----
> >>> From: Will Deacon <will@kernel.org>
> >>> Sent: Monday, November 23, 2020 8:44 AM
> >>> To: linux-kernel@vger.kernel.org
> >>> Cc: linux-pci@vger.kernel.org; iommu@lists.linux-foundation.org;=20
> >>> Will Deacon <will@kernel.org>; Bjorn Helgaas=20
> >>> <bhelgaas@google.com>; Deucher, Alexander=20
> >>> <Alexander.Deucher@amd.com>; Edgar Merger=20
> >>> <Edgar.Merger@emerson.com>; Joerg Roedel <jroedel@suse.de>
> >>> Subject: [PATCH] PCI: Mark AMD Raven iGPU ATS as broken
> >>>
> >>> Edgar Merger reports that the AMD Raven GPU does not work reliably=20
> >>> on his system when the IOMMU is enabled:
> >>>
> >>>    | [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx timeout,=20
> >>> signaled seq=3D1, emitted seq=3D3
> >>>    | [...]
> >>>    | amdgpu 0000:0b:00.0: GPU reset begin!
> >>>    | AMD-Vi: Completion-Wait loop timed out
> >>>    | iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT
> >>> device=3D0b:00.0 address=3D0x38edc0970]
> >>>
> >>> This is indicative of a hardware/platform configuration issue so,=20
> >>> since disabling ATS has been shown to resolve the problem, add a=20
> >>> quirk to match this particular device while Edgar follows-up with AMD=
 for more information.
> >>>
> >>> Cc: Bjorn Helgaas <bhelgaas@google.com>
> >>> Cc: Alex Deucher <alexander.deucher@amd.com>
> >>> Reported-by: Edgar Merger <Edgar.Merger@emerson.com>
> >>> Suggested-by: Joerg Roedel <jroedel@suse.de>
> >>> Link:
> >>> https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__lore&d=3DDwIDA=
w&c=3DjOURTkCZzT8tVB5xPEYIm3YJGoxoTaQsQPzPKJGaWbo&r=3DBJxhacqqa4K1PJGm6_-86=
2rdSP13_P6LVp7j_9l1xmg&m=3DlNXu2xwvyxEZ3PzoVmXMBXXS55jsmfDicuQFJqkIOH4&s=3D=
_5VDNCRQdA7AhsvvZ3TJJtQZ2iBp9c9tFHIleTYT_ZM&e=3D .
> >>> kernel.org/linux-
> >>> iommu/MWHPR10MB1310F042A30661D4158520B589FC0@MWHPR10M
> >>> B1310.namprd10.prod.outlook.com
> >>> her%40amd.com%7C1a883fe14d0c408e7d9508d88fb5df4e%7C3dd8961fe488
> >>> 4e608e11a82d994e183d%7C0%7C0%7C637417358593629699%7CUnknown%7
> >>> CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwi
> >>> LCJXVCI6Mn0%3D%7C1000&amp;sdata=3DTMgKldWzsX8XZ0l7q3%2BszDWXQJJ
> >>> LOUfX5oGaoLN8n%2B8%3D&amp;reserved=3D0
> >>> Signed-off-by: Will Deacon <will@kernel.org>
> >>> ---
> >>>
> >>> Hi all,
> >>>
> >>> Since Joerg is away at the moment, I'm posting this to try to make=20
> >>> some progress with the thread in the Link: tag.
> >> + Felix
> >>
> >> What system is this?  Can you provide more details?  Does a sbios=20
> >> update fix this?  Disabling ATS for all Ravens will break GPU=20
> >> compute for a lot of people.  I'd prefer to just black list this=20
> >> particular system (e.g., just SSIDs or revision) if possible.
>=20
> +Ray
>=20
> There are already many systems where the IOMMU is disabled in the=20
> BIOS, or the CRAT table reporting the APU compute capabilities is=20
> broken. Ray has been working on a fallback to make APUs behave like=20
> dGPUs on such systems. That should also cover this case where ATS is=20
> blacklisted. That said, it affects the programming model, because we=20
> don't support the unified and coherent memory model on dGPUs like we=20
> do on APUs with IOMMUv2. So it would be good to make the conditions=20
> for this workaround as narrow as possible.

Yes, besides the comments from Alex and Felix, may we get your firmware ver=
sion (SMC firmware which is from SBIOS) and device id?

> >>>    | [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx timeout,=20
> >>> signaled seq=3D1, emitted seq=3D3

It looks only gfx ib test passed, and fails to lanuch desktop, am I right?

We would like to see whether it is Raven, Raven kicker (new Raven), or Pica=
sso. In our side, per the internal test result, we didn't see the similiar =
issue on Raven kicker and Picasso platform.

Thanks,
Ray

>=20
> These are the relevant changes in KFD and Thunk for reference:
>=20
> ### KFD ###
>=20
> commit 914913ab04dfbcd0226ecb6bc99d276832ea2908
> Author: Huang Rui <ray.huang@amd.com>
> Date:=A0=A0 Tue Aug 18 14:54:23 2020 +0800
>=20
>  =A0=A0=A0 drm/amdkfd: implement the dGPU fallback path for apu (v6)
>=20
>  =A0=A0=A0 We still have a few iommu issues which need to address, so for=
ce=20
> raven
>  =A0=A0=A0 as "dgpu" path for the moment.
>=20
>  =A0=A0=A0 This is to add the fallback path to bypass IOMMU if IOMMU v2 i=
s=20
> disabled
>  =A0=A0=A0 or ACPI CRAT table not correct.
>=20
>  =A0=A0=A0 v2: Use ignore_crat parameter to decide whether it will go wit=
h=20
> IOMMUv2.
>  =A0=A0=A0 v3: Align with existed thunk, don't change the way of raven, o=
nly=20
> renoir
>  =A0=A0=A0=A0=A0=A0=A0 will use "dgpu" path by default.
>  =A0=A0=A0 v4: don't update global ignore_crat in the driver, and revise=
=20
> fallback
>  =A0=A0=A0=A0=A0=A0=A0 function if CRAT is broken.
>  =A0=A0=A0 v5: refine acpi crat good but no iommu support case, and renam=
e=20
> the
>  =A0=A0=A0=A0=A0=A0=A0 title.
>  =A0=A0=A0 v6: fix the issue of dGPU initialized firstly, just modify the=
=20
> report
>  =A0=A0=A0=A0=A0=A0=A0 value in the node_show().
>=20
>  =A0=A0=A0 Signed-off-by: Huang Rui <ray.huang@amd.com>
>  =A0=A0=A0 Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>
>  =A0=A0=A0 Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
>=20
> ### Thunk ###
>=20
> commit e32482fa4b9ca398c8bdc303920abfd672592764
> Author: Huang Rui <ray.huang@amd.com>
> Date:=A0=A0 Tue Aug 18 18:54:05 2020 +0800
>=20
>  =A0=A0=A0 libhsakmt: remove is_dgpu flag in the hsa_gfxip_table
>=20
>  =A0=A0=A0 Whether use dgpu path will check the props which exposed from =
kernel.
>  =A0=A0=A0 We won't need hard code in the ASIC table.
>=20
>  =A0=A0=A0 Signed-off-by: Huang Rui <ray.huang@amd.com>
>  =A0=A0=A0 Change-Id: I0c018a26b219914a41197ff36dbec7a75945d452
>=20
> commit 7c60f6d912034aa67ed27b47a29221422423f5cc
> Author: Huang Rui <ray.huang@amd.com>
> Date:=A0=A0 Thu Jul 30 10:22:23 2020 +0800
>=20
>  =A0=A0=A0 libhsakmt: implement the method that using flag which exposed =
by=20
> kfd to configure is_dgpu
>=20
>  =A0=A0=A0 KFD already implemented the fallback path for APU. Thunk will =
use=20
> flag
>  =A0=A0=A0 which exposed by kfd to configure is_dgpu instead of hardcode =
before.
>=20
>  =A0=A0=A0 Signed-off-by: Huang Rui <ray.huang@amd.com>
>  =A0=A0=A0 Change-Id: I445f6cf668f9484dd06cd9ae1bb3cfe7428ec7eb
>=20
> Regards,
>  =A0 Felix
>=20
>=20
> > Cheers, Alex. I'll have to defer to Edgar for the details, as my=20
> > understanding from the original thread over at:
> >
> > https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__lore.kernel.org
> > _linux-2Diommu_MWHPR10MB1310CDB6829DDCF5EA84A14689150-40MWHPR10MB131
> > 0.namprd10.prod.outlook.com_&d=3DDwIDAw&c=3DjOURTkCZzT8tVB5xPEYIm3YJGox=
o
> > TaQsQPzPKJGaWbo&r=3DBJxhacqqa4K1PJGm6_-862rdSP13_P6LVp7j_9l1xmg&m=3DlNX=
u
> > 2xwvyxEZ3PzoVmXMBXXS55jsmfDicuQFJqkIOH4&s=3DdsAVVJbD7gJIj3ctZpnnU60y21
> > ijWZmZ8xmOK1cO_O0&e=3D
> >
> > is that this is a board developed by his company.
> >
> > Edgar -- please can you answer Alex's questions?
> >
> > Will