Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Thu, 25 Jan 2018 09:54:21 +0100 (CET)
From:   Thomas Gleixner <tglx@linutronix.de>
To:     Lyude Paul <lyude@redhat.com>
cc:     "Ghannam, Yazen" <Yazen.Ghannam@amd.com>,
        "hpa@zytor.com" <hpa@zytor.com>,
        "keith.busch@intel.com" <keith.busch@intel.com>,
        "mingo@kernel.org" <mingo@kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Borislav Petkov <bp@alien8.de>
Subject: Re: "irq/matrix: Spread interrupts on allocation" breaks nouveau in
 mainline kernel
In-Reply-To: <1516823810.4109.26.camel@redhat.com>
Message-ID: <alpine.DEB.2.20.1801250945080.2020@nanos>
References: <1516744873.29151.3.camel@redhat.com>  <1516757219.29151.7.camel@redhat.com>  <alpine.DEB.2.20.1801241351080.1997@nanos>  <1516816150.4109.2.camel@redhat.com>  <DM5PR12MB19161E66F534B7050B8CCBF0F8E20@DM5PR12MB1916.namprd12.prod.outlook.com>
 <1516823810.4109.26.camel@redhat.com>
User-Agent: Alpine 2.20 (DEB 67 2015-01-07)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Wed, 24 Jan 2018, Lyude Paul wrote:
> Sorry about that! Let me clarify a little bit: this is a problem that shows up
> on mainline. Normally when we suspend the GPU in nouveau, we free the IRQs
> it's using before going into suspend
> (drivers/gpu/drm/nouveau/nvkm/subdev/pci/base.c:88), then reserve IRQs again
> on resume (drivers/gpu/drm/nouveau/nvkm/subdev/pci/base.c:134). Since this
> patch got pushed to mainline, the IRQ we get from request_irq() ends up having
> the same MSI vector as another device on the system:

It's not the same.

>     nouveau:
>      parent:
>         domain:  VECTOR
>          hwirq:   0x2f
>          chip:    APIC
>           flags:   0x0
>          Vector:    35
>          Target:     1

Vector 35 on CPU1

>     After resume and allocating the interrupt for nouveau again, we get a message
>     from the kernel saying: 
> 
>     [  217.150787] do_IRQ: 1.35 No irq handler for vector

That's because there is a pending irq on the old vector for unknown reasons.

>     As well, nouveau ends up getting no interrupts from the card and as a result
>     fails to come back up:
> 
>     [  219.153049] nouveau 0000:22:00.0: DRM: EVO timeout
>     [  220.226254] r8169 0000:1e:00.0 enp30s0: link up
>     [  221.153054] nouveau 0000:22:00.0: DRM: base-0: timeout
>     [  223.153528] nouveau 0000:22:00.0: DRM: base-0: timeout
> 
>     If we look through all of the other IRQ allocations, we'll find that now two
>     devices have the MSI vector 35:
> 
>     nouveau:
>      parent:
>         domain:  VECTOR
>          hwirq:   0x2f
>          chip:    APIC
>           flags:   0x0
>          Vector:    35
>          Target:     1

Vector 35 on CPU1

>     and the PCI bridge (00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD]
>     Family 17h (Models 00h-0fh) PCIe GPP Bridge):
> 
>          parent:
>             domain:  VECTOR
>              hwirq:   0x19
>              chip:    APIC
>               flags:   0x0
>              Vector:    35
>              Target:     0

Vector 35 on CPU0. Same vector but different CPUs. So it's NOT the same
thing.

The real issue is something completely different and the revert of this
patch merily papers over the underlying problem. I'm pretty sure that you
can trigger this even with the revert in place. Do the following before
suspend:

    echo 2 >/proc/irq/$NOUVEAUIRQ/smp_affinity_list

Then do suspend/resume and you should end up with the same situation.

I can't tell from your dmesg, but I'm pretty confident that

>     [  217.150787] do_IRQ: 1.35 No irq handler for vector

happens _before_ the nouveau driver requests the irq again. Can please you
add some printk to the code in question to verify that?

Thanks,

	tglx