Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Sat, 27 Mar 2021 21:45:54 +0000
Message-ID: <87mtuofey5.wl-maz@kernel.org>
From:   Marc Zyngier <maz@kernel.org>
To:     Pali =?UTF-8?B?Um9ow6Fy?= <pali@kernel.org>
Cc:     Jianjun Wang <jianjun.wang@mediatek.com>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Rob Herring <robh+dt@kernel.org>,
        Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
        Ryder Lee <ryder.lee@mediatek.com>,
        Philipp Zabel <p.zabel@pengutronix.de>,
        Matthias Brugger <matthias.bgg@gmail.com>,
        linux-pci@vger.kernel.org, linux-mediatek@lists.infradead.org,
        devicetree@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-arm-kernel@lists.infradead.org, youlin.pei@mediatek.com,
        chuanjia.liu@mediatek.com, qizhong.cheng@mediatek.com,
        sin_jieyang@mediatek.com, drinkcat@chromium.org,
        Rex-BC.Chen@mediatek.com, anson.chuang@mediatek.com,
        Krzysztof Wilczyski <kw@linux.com>
Subject: Re: [v9,5/7] PCI: mediatek-gen3: Add MSI support
In-Reply-To: <20210327202904.nvn7tfodmc2xw23l@pali>
References: <20210324030510.29177-1-jianjun.wang@mediatek.com>
        <20210324030510.29177-6-jianjun.wang@mediatek.com>
        <20210327192837.4rr46oeiuokritlc@pali>
        <87o8f4fkkh.wl-maz@kernel.org>
        <20210327202904.nvn7tfodmc2xw23l@pali>
User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue)
 FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1
 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO)
MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue")
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8BIT
Precedence: bulk

On Sat, 27 Mar 2021 20:29:04 +0000,
Pali Rohár <pali@kernel.org> wrote:
> 
> On Saturday 27 March 2021 19:44:30 Marc Zyngier wrote:
> > On Sat, 27 Mar 2021 19:28:37 +0000,
> > Pali Rohár <pali@kernel.org> wrote:
> > > 
> > > On Wednesday 24 March 2021 11:05:08 Jianjun Wang wrote:
> > > > +static void mtk_pcie_msi_handler(struct mtk_pcie_port *port, int set_idx)
> > > > +{
> > > > +	struct mtk_msi_set *msi_set = &port->msi_sets[set_idx];
> > > > +	unsigned long msi_enable, msi_status;
> > > > +	unsigned int virq;
> > > > +	irq_hw_number_t bit, hwirq;
> > > > +
> > > > +	msi_enable = readl_relaxed(msi_set->base + PCIE_MSI_SET_ENABLE_OFFSET);
> > > > +
> > > > +	do {
> > > > +		msi_status = readl_relaxed(msi_set->base +
> > > > +					   PCIE_MSI_SET_STATUS_OFFSET);
> > > > +		msi_status &= msi_enable;
> > > > +		if (!msi_status)
> > > > +			break;
> > > > +
> > > > +		for_each_set_bit(bit, &msi_status, PCIE_MSI_IRQS_PER_SET) {
> > > > +			hwirq = bit + set_idx * PCIE_MSI_IRQS_PER_SET;
> > > > +			virq = irq_find_mapping(port->msi_bottom_domain, hwirq);
> > > > +			generic_handle_irq(virq);
> > > > +		}
> > > > +	} while (true);
> > > 
> > > Hello!
> > > 
> > > Just a question, cannot this while-loop cause block of processing other
> > > interrupts?
> > 
> > This is a level interrupt. You don't have much choice but to handle it
> > immediately, although an alternative would be to mask it and deal with
> > it in a thread. And since Linux doesn't deal with interrupt priority,
> > a screaming interrupt is never a good thing.
> 
> I see. Something like "interrupt priority" (which does not exist?) would
> be needed to handle it.

Interrupt priorities definitely exist, but Linux doesn't use
them. Furthermore, This wouldn't be relevant here as you get a bunch
of MSI multiplexed onto a single one. Where would you apply the
priority?

> 
> > > I have done tests with different HW (aardvark) but with same while(true)
> > > loop logic. One XHCI PCIe controller was sending MSI interrupts too fast
> > > and interrupt handler with this while(true) logic was in infinite loop.
> > > During one IRQ it was calling infinite many times generic_handle_irq()
> > > as HW was feeding new and new MSI hwirq into status register.
> > 
> > Define "too fast".
> 
> Fast - next interrupt comes prior checking if while(true)-loop should stop.

That's definitely not something you can easily fix at the interrupt
handling level. You need to prevent this from happening. That's
usually the result of a misprogramming or a HW bug.

> > If something in the system is able to program the
> > XHCI device in such a way that it causes a screaming interrupt, that's
> > the place to look for problems, and probably not in the interrupt
> > handling itself, which does what it is supposed to do.
> > 
> > > But this is different HW, so it can have different behavior and does not
> > > have to cause above issue.
> > > 
> > > I have just spotted same code pattern for processing MSI interrupts...
> > 
> > This is a common pattern that you will find in pretty much any
> > interrupt handling/demuxing, and is done this way when the cost of
> > taking the exception is high compared to that of handling it.
> 
> And would not help if while(true)-loop is replaced by loop with upper
> limit of iterations? Or just call only one iteration?

That wouldn't change much: you would still have the interrupt being
pending, and it would fire again at the earliest opportunity.

At best, the root interrupt controller is able to present you with
another interrupt before forcing you to deal with the one you have
ignored again. But you cannot rely on that either.

And to be honest, other interrupts are only a part of the problem you
are describing. With a screaming interrupt, you can't execute
userspace. This is as bad as it gets.

	M.

-- 
Without deviation from the norm, progress is not possible.