Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1752294AbeAQJcQ (ORCPT <rfc822;ralf@linux-mips.org> + 1 other);
        Wed, 17 Jan 2018 04:32:16 -0500
Received: from Galois.linutronix.de ([146.0.238.70]:45370 "EHLO
        Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1750810AbeAQJcO (ORCPT
        <rfc822;linux-kernel@vger.kernel.org>);
        Wed, 17 Jan 2018 04:32:14 -0500
Date: Wed, 17 Jan 2018 10:32:12 +0100 (CET)
From: Thomas Gleixner <tglx@linutronix.de>
To: Keith Busch <keith.busch@intel.com>
cc: LKML <linux-kernel@vger.kernel.org>
Subject: Re: [BUG 4.15-rc7] IRQ matrix management errors
In-Reply-To: <alpine.DEB.2.20.1801171020440.1777@nanos>
Message-ID: <alpine.DEB.2.20.1801171030150.1777@nanos>
References: <20180115025759.GG13580@localhost.localdomain> <20180115030255.GA13921@localhost.localdomain> <alpine.DEB.2.20.1801151012230.1810@nanos> <20180116061641.GB32639@localhost.localdomain> <20180116071145.GA5643@localhost.localdomain>
 <alpine.DEB.2.20.1801161131160.1823@nanos> <alpine.DEB.2.20.1801161217300.1823@nanos> <20180117022511.GD6259@localhost.localdomain> <alpine.DEB.2.20.1801170832460.2366@nanos> <20180117075500.GB7562@localhost.localdomain>
 <alpine.DEB.2.20.1801171020440.1777@nanos>
User-Agent: Alpine 2.20 (DEB 67 2015-01-07)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Return-Path: <linux-kernel-owner@vger.kernel.org>

On Wed, 17 Jan 2018, Thomas Gleixner wrote:

> On Wed, 17 Jan 2018, Keith Busch wrote:
> > On Wed, Jan 17, 2018 at 08:34:22AM +0100, Thomas Gleixner wrote:
> > > Can you trace the matrix allocations from the very beginning or tell me how
> > > to reproduce. I'd like to figure out why this is happening.
> > 
> > Sure, I'll get the irq_matrix events.
> > 
> > I reproduce this on a machine with 112 CPUs and 3 NVMe controllers. The
> > first two NVMe want 112 MSI-x vectors, and the last only 31 vectors. The
> > test runs 'modprobe nvme' and 'modprobe -r nvme' in a loop with 10
> > second delay between each step. Repro occurs within a few iterations,
> > sometimes already broken after the initial boot.
> 
> That doesn't sound right. The vectors should be spread evenly accross the
> CPUs. So ENOSPC should never happen.
> 
> Can you please take snapshots of /sys/kernel/debug/irq/ between the
> modprobe and modprobe -r steps?

The allocation fails because CPU1 has exhausted it's vector space here:

[002] d...   333.028216: irq_matrix_alloc_managed: bit=34 cpu=1 online=1 avl=0 alloc=202 managed=2 online_maps=112 global_avl=22085, global_rsvd=158, total_alloc=460

Now the interesting question is how that happens.

Thanks,

	tglx