Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Wed, 8 Jan 2020 14:15:11 -0600
From:   Bjorn Helgaas <helgaas@kernel.org>
To:     Muni Sekhar <munisekharrms@gmail.com>
Cc:     linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: pcie: xilinx: kernel hang - ISR readl()
Message-ID: <20200108201511.GA195980@google.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <CAHhAz+ijBTp55gZYAejWthnvdmR_qyQJpVV4r1gyQ-Kud6t9qg@mail.gmail.com>
User-Agent: Mutt/1.10.1 (2018-07-13)
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Tue, Jan 07, 2020 at 09:45:13PM +0530, Muni Sekhar wrote:
> Hi,
> 
> I have module with Xilinx FPGA. It implements UART(s), SPI(s),
> parallel I/O and interfaces them to the Host CPU via PCI Express bus.
> I see that my system freezes without capturing the crash dump for
> certain tests. I debugged this issue and it was tracked down to the
> below mentioned interrupt handler code.
> 
> 
> In ISR, first reads the Interrupt Status register using ‘readl()’ as
> given below.
>     status = readl(ctrl->reg + INT_STATUS);
> 
> 
> And then clears the pending interrupts using ‘writel()’ as given blow.
>         writel(status, ctrl->reg + INT_STATUS);
> 
> 
> I've noticed a kernel hang if INT_STATUS register read again after
> clearing the pending interrupts.
> 
> Can someone clarify me why the kernel hangs without crash dump incase
> if I read the INT_STATUS register using readl() after clearing the
> pending bits?
> 
> Can readl() block?

readl() should not block in software.  Obviously at the hardware CPU
instruction level, the read instruction has to wait for the result of
the read.  Since that data is provided by the device, i.e., your FPGA,
it's possible there's a problem there.

Can you tell whether the FPGA has received the Memory Read for
INT_STATUS and sent the completion?

On the architectures I'm familiar with, if a device doesn't respond,
something would eventually time out so the CPU doesn't wait forever.

> Snippet of the ISR code is given blow:
> 
> https://pastebin.com/WdnZJZF5
> 
> 
> 
> static irqreturn_t pcie_isr(int irq, void *dev_id)
> 
> {
> 
>         struct test_device *ctrl = data;
> 
>         u32 status;
> 
> …
> 
> 
> 
>         status = readl(ctrl->reg + INT_STATUS);
> 
>         /*
> 
>          * Check to see if it was our interrupt
> 
>          */
> 
>         if (!(status & 0x000C))
> 
>                 return IRQ_NONE;
> 
> 
> 
>         /* Clear the interrupt */
> 
>         writel(status, ctrl->reg + INT_STATUS);
> 
> 
> 
>         if (status & 0x0004) {
> 
>                 /*
> 
>                  * Tx interrupt pending.
> 
>                  */
> 
>                  ....
> 
>        }
> 
> 
> 
>         if (status & 0x0008) {
> 
>                 /* Rx interrupt Pending */
> 
>                 /* The system freezes if I read again the INT_STATUS
> register as given below */
> 
>                 status = readl(ctrl->reg + INT_STATUS);
> 
>                 ....
> 
>         }
> 
> ..
> 
>         return IRQ_HANDLED;
> }
> 
> 
> 
> -- 
> Thanks,
> Sekhar