Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 23 Aug 2022 13:46:36 +0200
Message-ID: <87ilmjqj1f.wl-tiwai@suse.de>
From:   Takashi Iwai <tiwai@suse.de>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Lu Baolu <baolu.lu@linux.intel.com>,
        Joerg Roedel <jroedel@suse.de>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Robin Murphy <robin.murphy@arm.com>,
        Eric Auger <eric.auger@redhat.com>,
        regressions@lists.linux.dev, linux-kernel@vger.kernel.org
Subject: Re: [REGRESSION 5.19.x] AMD HD-audio devices missing on 5.19
In-Reply-To: <87h723sdde.wl-tiwai@suse.de>
References: <874jy4cqok.wl-tiwai@suse.de>
        <20220823010021.GA5967@nvidia.com>
        <87h723sdde.wl-tiwai@suse.de>
User-Agent: Wanderlust/2.15.9 (Almost Unreal) Emacs/27.2 Mule/6.0
MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue")
Content-Type: text/plain; charset=US-ASCII
Precedence: bulk

On Tue, 23 Aug 2022 08:06:05 +0200,
Takashi Iwai wrote:
> 
> On Tue, 23 Aug 2022 03:00:21 +0200,
> Jason Gunthorpe wrote:
> > 
> > On Mon, Aug 22, 2022 at 04:12:59PM +0200, Takashi Iwai wrote:
> > > Hi,
> > > 
> > > we've received regression reports about the missing HD-audio devices
> > > on AMD platforms, and this turned out to be caused by the commit
> > > 512881eacfa72c2136b27b9934b7b27504a9efc2
> > >     bus: platform,amba,fsl-mc,PCI: Add device DMA ownership management
> > > 
> > > The details are found in openSUSE bugzilla:
> > >   https://bugzilla.suse.com/show_bug.cgi?id=1202492
> > > 
> > > The problem seems to be that HD-audio (both onboard analog and HDMI)
> > > PCI devices are assigned to the same IOMMU group as AMD graphics PCI
> > > device, and once after the AMDGPU is initialized beforehand, those
> > > audio devices can't be probed since iommu_device_use_default_domain()
> > > returns -EBUSY.
> > 
> > Can you describe exactly what drivers are involved in this? If it is
> > the above commit then several devices are sharing an iommu group and
> > one of them (well, the only one already attached, I suppose) has made
> > the group unsharable.
> > 
> > With grep I don't see an obvious place where the AMDGPU driver would
> > mess with the iommu configuration, so I have no guess.
> 
> I have also no concrete clue, either :)
> At least, drivers/gpu/drm/amd/amdkfd/kfd_iommu.c calls
> amd_iommu_init_device(), and this invokes iommu_attach_group(), which
> may change group->domain.  But it was just my wild guess, and it might
> be others, indeed.
> 
> > It would be good to have some debugging to confirm if it is
> > group->owner (should be impossible, suggests memory corruption if it
> > is) or group->domain != group->default_domain.
> >
> > Most likely it is the later, but I can't see how that could happen on
> > a system like this.. There is no obvious manipulation in AMDGPU, for
> > instance.
> > 
> > So debugging to find the backtrace for exactly when 
> >  group->domain != group->default_domain
> > Occurs for the troubled group would be necessary.
> 
> OK, will try to build a test kernel with some debug prints and ask the
> reporters.  It may take some time.

It was tested now and confirmed that the call path is via AMDGPU, as
expected:
  amdgpu_pci_probe ->
  amdgpu_driver_load_kms ->
  amdgpu_device_init ->
  amdgpu_amdkfd_device_init ->
  kgd2kfd_device_init ->
  kgd2kfd_resume_iommu ->
  kfd_iommu_resume ->
  amd_iommu_init_device ->
  iommu_attach_group ->
  __iommu_attach_group

At first AMDGPU driver is probed, and the iommu_attach_group() call
above changes the assigned group->domain.  Afterwards, when HD-audio
devices are probed, it fails because:
- Both HD-audio PCI devices belong to the very same IOMMU group as the
  AMD graphics PCI device
- PCI core calls iommu_device_use_default_domain() and the check
  fails there because group->domain != group->default_domain

Takashi