Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <525730fd-5982-fea7-b6d5-2da69f225f04@amd.com>
Date:   Tue, 10 Jan 2023 21:38:13 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.8.1
Subject: =?UTF-8?Q?Re=3a_=5bregression=2c_bisected=2c_pci/iommu=5d_Bug=c2=a0?=
 =?UTF-8?Q?216865_-_Black_screen_when_amdgpu_started_during_6=2e2-rc1_boot_w?=
 =?UTF-8?Q?ith_AMD_IOMMU_enabled?=
Content-Language: en-US
To:     Matt Fagnani <matt.fagnani@bell.net>,
        Baolu Lu <baolu.lu@linux.intel.com>,
        Thorsten Leemhuis <regressions@leemhuis.info>
Cc:     Joerg Roedel <jroedel@suse.de>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        LKML <linux-kernel@vger.kernel.org>,
        "regressions@lists.linux.dev" <regressions@lists.linux.dev>,
        Linux PCI <linux-pci@vger.kernel.org>,
        Bjorn Helgaas <bhelgaas@google.com>
References: <15d0f9ff-2a56-b3e9-5b45-e6b23300ae3b@leemhuis.info>
 <5aa0e698-f715-0481-36e5-46505024ebc1@bell.net>
 <aea57c5f-2d20-c589-ad44-a63f1133a3db@linux.intel.com>
 <157c4ca4-370a-5d7e-fe32-c64d934f6979@amd.com>
 <223ee6d6-70ea-1d53-8bc2-2d22201d8dde@bell.net>
 <6fff9d10-f77f-e55a-9020-8a1bd34cf508@amd.com>
 <ff26929d-9fb0-3c85-2594-dc2937c1ba9a@bell.net>
From:   Vasant Hegde <vasant.hegde@amd.com>
In-Reply-To: <ff26929d-9fb0-3c85-2594-dc2937c1ba9a@bell.net>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?aXdXZDJIMHl2VE0wM2N5VStqVzZDQjBWWGxjUnJ5OUQzaHRLYmQwZWZYM3Fi?=
 =?utf-8?B?ZG8vSHFSVlJZTmZ0WHZDR24wR3BENU5HZzlKcFJtSkxoY1c0eHFEeGtTRTlM?=
 =?utf-8?B?RzFqMm9Sd1NXc0gva2dPNXhXNmx1ZWdpWlMzWGdacjl5dVVLZjN2VHJmcGU1?=
 =?utf-8?B?S01ReWZVYVVES29TSGpsdE5STmowbEdIMURjeC9IZVFVa01iZ2tZdWY5OGJk?=
 =?utf-8?B?UzhYZEZQWHBHRWs5M1d0YmNoSXh6SVJJWTUvMGVWQVlGRGZ4N3d2amZ3L2p5?=
 =?utf-8?B?Y09haFA2RzdJcHZ1VmxRWSthZHNVc21WUi9PY1p6Y3RCVFRvMUpYTGg2bjhQ?=
 =?utf-8?B?aUVEVGthS2xMTVFETjJ2MTl0OEliYjlZbVNzRzRicnYrZE1ONk92WUw0amo0?=
 =?utf-8?B?Tm0wdkFnUWt3KzMwT0VEb0ZmTFY1Z2UvMnF2VUpMREkrdUlvYzAvaktqbkdE?=
 =?utf-8?B?SG5YWjJ5L1FPT09sQVlhU3ZGRXNDUUd4VFRQZDZ3dmUvZzdEQTlHL052U0Rt?=
 =?utf-8?B?Q0FQSmxjL3dNUlFHK25tQ2hOWVlOZVZ3ckN3c1ovdS8rVkhuZnpHelQvdVhl?=
 =?utf-8?B?Zld4cDllNWtIUU5DaWMzVG9VMlF5QlZLS2p5ZU0wT2ZSeGw1eVZUTjdubzFI?=
 =?utf-8?B?d3BlQWx5K3ZrYnlzYWdhUFg4UFBLaTdsdFRiTkhUNFdpeGFJSThDRVpwY3l4?=
 =?utf-8?B?OEE5c3RERVROSEdINGNLano0K0FDcmNDRFhURmp5UWp3OUV5RmRIdWh4Vno1?=
 =?utf-8?B?cTJLRmpHQ0JhR0FqUmIxZzE0bU5jUVc3OW9HUUphaEFTZlRrQlFSaTByNzZT?=
 =?utf-8?B?NUI4MWpIWUxVT3NheE1LOEQ0Q0lkaDdkaDI5VE5MUDlkbjFMaC8rYnVzRUhl?=
 =?utf-8?B?WHE2MEgycjQ0cWJ1TUNmSDN5V28zbTBZYkpoK3pwKzN4Tll0bEVEcEZodTVO?=
 =?utf-8?B?Z3lvaWpuQUs2c3dGTktEOW4wUWpZbitxMVE2OHZmdWpLdFpobnREdnVYVWhQ?=
 =?utf-8?B?b0V1cEpYMWFndWE2bU43Qkdxc0twMzhHMDJDQ3pKeE92R0FDYkl1eHZtdFdB?=
 =?utf-8?B?ZzBOQS9Ta3hXY1g2UFZZRGJrdG1hZEowQlZNUG9uMnN4UlJvaHJKVGRrSXVV?=
 =?utf-8?B?dXdGVFlPZjBMWk5heDVTdHhPRFlFSzNjTDljYWxPSEYwZlFZTDRHTVJmOFo5?=
 =?utf-8?B?NUYvQ3ppRWNSS05xeWRFNU9kZUJrWEczRkZDMGFmRGs2eCtGL1l4blFxTVhN?=
 =?utf-8?B?N3puY3B0L3QzdHJDS3RmR0o4MXdaakJKWmtBTCtBeHZCK1VQMlNYbDN1WlNF?=
 =?utf-8?B?TmRmYzNhRjVhWHlOZXpoNjNydFVQUjRUaDA2anB5cXFRWURZNVdiOTFodzds?=
 =?utf-8?B?QkNSL1BiUzR2emxuVm9YWjhlU0lIUmMvQllOU3RpZEpaRmZjV3kwd0c3clY1?=
 =?utf-8?B?SWh5L1F6UXdSbmFGVzdwaHRCdWxxc0ltQ2ZnY2ttdEpLOE5jL0k3N3ZQU2hl?=
 =?utf-8?B?eHZZZmJ1cW1ja01MQkdWTzNjbmpjS1dpa09vZ3F6S1h4QitSbDdlbFJJdEtS?=
 =?utf-8?B?MWdiSXRRMVNwdlJRc3d3dUNiVGh4YkpLeVlvZitJT0h6ZzNFVkhRVVFRT3Jn?=
 =?utf-8?B?N3Rqc09kT1J1UWttU2VLRGdGVmNjNXFacW5wMGN6RXVOaUxvaWgybTc2MHdY?=
 =?utf-8?B?dmZMYXZEUm9oTzc3RmdwUXcrVXVHdFlTUW9keXhQUThuekpjY05DNGtDNHJV?=
 =?utf-8?B?R01IWWo3b2NYYU9aaGh6YWlVeUJLZURuNHpGOFk0S1FIZForRU9JdnhCQWtq?=
 =?utf-8?B?ZkQ3SytyR2FBYmdoZUJwSXUwNjUyU3YrNVNCWk5CUUxlcUxhQXdxMUJwMEJR?=
 =?utf-8?B?MXhPVmhuQnU4eC9mVkp4aDlzZE9mMVpzeTdiT0dSWjRWK2tWdG1BYnd2Tmd2?=
 =?utf-8?B?RjFNb1MrQ1BkNUVNY0QwUTRVVmR5Snc0bVlmZDdPdE0xbUdFaUxsNXRqQW1Y?=
 =?utf-8?B?UVBjU3F2L1JYOWdxR2lXODJLVThUOTNpYmJiTU9UU01OR0JZeGNheFJveEpY?=
 =?utf-8?B?NS9TdXQzVklGLytIQ04raGV6LzFwZStMcjdTRDlIVnJqOVZtdlM1ejZDdnQw?=
 =?utf-8?B?VGFJM0t4TnM5cm8rYmp2N0FJeEs4Z0VVc1Y4SERpZFR2YnpZRE02NGpGbXFV?=
 =?utf-8?Q?fPsE9avORRNuu3+0bjsdF6+ZDrSEzKKZ2i9Aj7tCA3xf?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c13f972d-adc5-4fe9-5cb6-08daf324e82b
X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB6048.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Jan 2023 16:08:26.9780
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: s2zDj3r5HtXs3vykKX9HgFguyNAz67EyxYYsPzH0qDxr5jycZiSRezbhM7Ufhf3YM9wvlQs5wdnRsHCmdau32w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB7508
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Matt,


On 1/6/2023 12:58 PM, Matt Fagnani wrote:
> I booted 6.2-rc2 + patch with rd.driver.blacklist=amdgpu on the kernel command
> line to prevent amdgpu from being started while the initramfs was in use. The
> black screen problem happened later in the boot. I pressed sysrq+alt+s,u,b to do
> an emergency sync, remount read-only, and reboot. The journal for that boot was
> shown on the next boot. The two warnings which I previously reported weren't
> shown in the journal, but the same null pointer dereference which made amdgpu
> crash happened. I'm attaching the kernel log from the journal of that boot.
> 

Thanks for your effort to get boot log. This is helpful.

Looking into the code further,
  iommu_detach_group() didn't attach devices back to default_domain. So IOMMU
point of view device group was left in inconsistent state. This resulted in
IOMMU throwing page fault errors and amd IOMMU event handler code always assumes
that domain is setup properly. That resulted in below NULL pointer dereference
issue.

  Jan 06 02:07:52 kernel: BUG: kernel NULL pointer dereference, address:
0000000000000058
  Jan 06 02:07:52 kernel: #PF: supervisor read access in kernel mode
  Jan 06 02:07:53 kernel: #PF: error_code(0x0000) - not-present page
  Jan 06 02:07:53 kernel: PGD 0 P4D 0
  Jan 06 02:07:53 kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI
  Jan 06 02:07:53 kernel: CPU: 2 PID: 56 Comm: irq/24-AMD-Vi Not tainted
6.2.0-rc2+ #89
  Jan 06 02:07:53 kernel: Hardware name: HP HP Laptop 15-bw0xx/8332, BIOS F.52
12/03/2019
  Jan 06 02:07:53 kernel: RIP: 0010:report_iommu_fault+0x11/0x90

Ideally if domain attach fails (in this case its because pasid capability check
returned error) we should put devices back to original domain.. so that it can
continue without PASID capability.

I have a patch to handle these error conditions (not the fix for original
issue). I will try to post it soon.

-Vasant