Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Tue, 28 Apr 2020 09:57:24 +0200
Message-ID: <s5hv9lkm49n.wl-tiwai@suse.de>
From:   Takashi Iwai <tiwai@suse.de>
To:     Alex Deucher <alexdeucher@gmail.com>
Cc:     Nicholas Johnson <nicholas.johnson-opensource@outlook.com.au>,
        "Zhou, David(ChunMing)" <David1.Zhou@amd.com>,
        "alsa-devel@alsa-project.org" <alsa-devel@alsa-project.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "amd-gfx@lists.freedesktop.org" <amd-gfx@lists.freedesktop.org>,
        Takashi Iwai <tiwai@suse.com>, Lukas Wunner <lukas@wunner.de>,
        "Deucher, Alexander" <Alexander.Deucher@amd.com>,
        "Koenig, Christian" <Christian.Koenig@amd.com>
Subject: Re: [PATCH 0/1] Fiji GPU audio register timeout when in BACO state
In-Reply-To: <CADnq5_MMQ5_MjEg=bkJJGMJP53RjB3yxvOW0nUDeWxzg3Q0pVQ@mail.gmail.com>
References: <PSXP216MB0438D2AF96CE0D4F83F48C4D80AE0@PSXP216MB0438.KORP216.PROD.OUTLOOK.COM>
        <MN2PR12MB4488E4909C1488FB507E0BF5F7AF0@MN2PR12MB4488.namprd12.prod.outlook.com>
        <s5ho8rdnems.wl-tiwai@suse.de>
        <PSXP216MB04387BF6B5F8DA84749E5D6F80AF0@PSXP216MB0438.KORP216.PROD.OUTLOOK.COM>
        <CADnq5_M=QEqxuCKjb_qZvFSvwM5eLEFfsepxYYXoouFoe5bn7A@mail.gmail.com>
        <s5h4kt4ojrf.wl-tiwai@suse.de>
        <CADnq5_MMQ5_MjEg=bkJJGMJP53RjB3yxvOW0nUDeWxzg3Q0pVQ@mail.gmail.com>
User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI/1.14.6 (Maruoka)
 FLIM/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL/10.8 Emacs/25.3
 (x86_64-suse-linux-gnu) MULE/6.0 (HANACHIRUSATO)
MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka")
Content-Type: text/plain; charset=US-ASCII
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Mon, 27 Apr 2020 20:43:54 +0200,
Alex Deucher wrote:
> 
> On Mon, Apr 27, 2020 at 2:39 PM Takashi Iwai <tiwai@suse.de> wrote:
> >
> > On Mon, 27 Apr 2020 20:28:12 +0200,
> > Alex Deucher wrote:
> > >
> > > On Mon, Apr 27, 2020 at 2:07 PM Nicholas Johnson
> > > <nicholas.johnson-opensource@outlook.com.au> wrote:
> > > >
> > > > On Mon, Apr 27, 2020 at 05:15:55PM +0200, Takashi Iwai wrote:
> > > > > On Mon, 27 Apr 2020 16:22:21 +0200,
> > > > > Deucher, Alexander wrote:
> > > > > >
> > > > > > [AMD Public Use]
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Nicholas Johnson <nicholas.johnson-opensource@outlook.com.au>
> > > > > > > Sent: Sunday, April 26, 2020 12:02 PM
> > > > > > > To: linux-kernel@vger.kernel.org
> > > > > > > Cc: Deucher, Alexander <Alexander.Deucher@amd.com>; Koenig, Christian
> > > > > > > <Christian.Koenig@amd.com>; Zhou, David(ChunMing)
> > > > > > > <David1.Zhou@amd.com>; Nicholas Johnson <nicholas.johnson-
> > > > > > > opensource@outlook.com.au>
> > > > > > > Subject: [PATCH 0/1] Fiji GPU audio register timeout when in BACO state
> > > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Since Linux v5.7-rc1 / commit 4fdda2e66de0 ("drm/amdgpu/runpm: enable
> > > > > > > runpm on baco capable VI+ asics"), my AMD R9 Nano has been using runpm /
> > > > > > > BACO. You can tell visually when it sleeps, because the fan on the graphics
> > > > > > > card is switched off to save power. It did not spin down the fan in v5.6.x.
> > > > > > >
> > > > > > > This is great (I love it), except that when it is sleeping, the PCIe audio function
> > > > > > > of the GPU has issues if anything tries to access it. You get dmesg errors such
> > > > > > > as these:
> > > > > > >
> > > > > > > snd_hda_intel 0000:08:00.1: spurious response 0x0:0x0, last cmd=0x170500
> > > > > > > snd_hda_intel 0000:08:00.1: azx_get_response timeout, switching to polling
> > > > > > > mode: last cmd=0x001f0500 snd_hda_intel 0000:08:00.1: No response from
> > > > > > > codec, disabling MSI: last cmd=0x001f0500 snd_hda_intel 0000:08:00.1: No
> > > > > > > response from codec, resetting bus: last cmd=0x001f0500
> > > > > > > snd_hda_codec_hdmi hdaudioC1D0: Unable to sync register 0x2f0d00. -11
> > > > > > >
> > > > > > > The above is with the Fiji XT GPU at 0000:08:00.0 in a Thunderbolt enclosure
> > > > > > > (not that Thunderbolt should affect it, but I feel I should mention it just in
> > > > > > > case). I dropped a lot of duplicate dmesg lines, as some of them repeated a
> > > > > > > lot of times before the driver gave up.
> > > > > > >
> > > > > > > I offer this patch to disable runpm for Fiji while a fix is found, if you decide
> > > > > > > that is the best approach. Regardless, I will gladly test any patches you come
> > > > > > > up with instead and confirm that the above issue has been fixed.
> > > > > > >
> > > > > > > I cannot tell if any other GPUs are affected. The only other cards to which I
> > > > > > > have access are a couple of AMD R9 280X (Tahiti XT), which use radeon driver
> > > > > > > instead of amdgpu driver.
> > > > > >
> > > > > > Adding a few more people.  Do you know what is accessing the audio?  The audio should have a dependency on the GPU device.  The GPU won't enter runtime pm until the audio has entered runtime pm and vice versa on resume. Please attach a copy of your dmesg output and lspci output.
> > > >
> > > > pci 0000:08:00.1: D0 power state depends on 0000:08:00.0
> > > > The above must be the dependency of which you speak from dmesg.
> > > >
> > > > Accessing the audio? I did not have a single method for triggering it.
> > > > Sometimes it happened on shutdown. Sometimes when restarting gdm.
> > > > Sometimes when playing with audio settings in Cinnamon Desktop. But most
> > > > often when changing displays. It might have something to do with the
> > > > audio device associated with a monitor being created when the monitor is
> > > > found. If an audio device is created, then pulseaudio might touch it.
> > > > Sorry, this is a very verbose "not quite sure".
> > > >
> > > > To trigger the bug, this time I did the following:
> > > >
> > > > 1. Boot laptop without Fiji and log in
> > > >
> > > > 2. Attach Fiji via Thunderbolt (no displays attached to Fiji) and
> > > > approve Thunderbolt device
> > > >
> > > > 3. Log in again because the session gets killed when GPU is hot-added
> > > >
> > > > 4. Wait for Fiji to fall asleep (fan stops)
> > > >
> > > > 5. Open "dmesg -w" on laptop display
> > > >
> > > > 6. Attach display to DisplayPort on Fiji (it should still stay asleep)
> > > >
> > > > 7. Do WindowsKey+P to activate external display. The error appears in
> > > > dmesg window that instant.
> > > >
> > > > Could it be a race condition when waking the card up?
> > > >
> > > > I cannot get the graphics card fan to spin down if the Thunderbolt
> > > > enclosure is attached at boot time. It only does it if hot-added.
> > > >
> > > > If you think it will help, I can take out the Fiji and put it in a test
> > > > rig and try to replicate the issue without Thunderbolt, but it looks
> > > > like it will not spin the fan down if Fiji is attached at boot time.
> > > >
> > > > Question, why would the fan not spin down if Fiji is attached at boot
> > > > time, and how would one make the said fan turn off? Aside from being
> > > > useful for pinning down the audio register issue, I would like to make
> > > > sure the power savings are realised whenever the GPU is not being used.
> > >
> > > Presumably something is using the device.  Maybe a framebuffer console
> > > or X?  Or maybe the something like tlp has disabled runtime pm on your
> > > device?  You can see the current status by reading the files in
> > > /sys/class/drm/cardX/device/power/ .  Replace cardX with card0, card1,
> > > etc. depending on which device is the radeon card.
> > >
> > > FWIW, I have a fiji board in a desktop system and it worked fine when
> > > this code was enabled.
> >
> > Is the new DC code used for Fiji boards?  IIRC, the audio component
> > binding from amdgpu is enabled only for DC, and without the audio
> > component binding the runtime PM won't be linked up, hence you can't
> > power up GPU from the audio side access automatically.
> >
> 
> Yes, DC is enabled by default for all cards with runtime pm enabled.

OK, thanks, I found that amdgpu got bound via component in the dmesg
output, too:

[   21.294927] snd_hda_intel 0000:08:00.1: bound 0000:08:00.0 (ops amdgpu_dm_audio_component_bind_ops [amdgpu])

This is the place soon after amdgpu driver gets initialized.
Then we see later another initialization phase:

[   26.904127] rfkill: input handler enabled
[   37.264152] [drm] PCIE GART of 1024M enabled (table at 0x000000F400000000).

here shows 10 seconds between them.  Then, it complained something:


[   37.363287] [drm] UVD initialized successfully.
[   37.473340] [drm] VCE initialized successfully.
[   37.477942] amdgpu 0000:08:00.0: [drm] Cannot find any crtc or sizes

... and go further, and hitting HD-audio error:


[   38.936624] [drm] fb mappable at 0x4B0696000
[   38.936626] [drm] vram apper at 0x4B0000000
[   38.936626] [drm] size 33177600
[   38.936627] [drm] fb depth is 24
[   38.936627] [drm]    pitch is 15360
[   38.936673] amdgpu 0000:08:00.0: fb1: amdgpudrmfb frame buffer device
[   40.092223] snd_hda_intel 0000:08:00.1: azx_get_response timeout, switching to polling mode: last cmd=0x00170500

After this point, HD-audio communication was screwed up.

This lastcmd in the above message is AC_SET_POWER_STATE verb for the
root node to D0, so the very first command to power up the codec. 
The rest commands are also about the power up of each node, so the
whole error indicate that the power up at runtime resume failed.

So, this looks to me as if the device gets runtime-resumed at the bad
moment?


thanks,

Takashi