Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <0f19cc67-ccb1-7cd1-5475-d2ec0e1abfc0@quicinc.com>
Date:   Thu, 4 Aug 2022 08:22:24 -0600
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.6.0
Subject: Re: [PATCH] PCI: hv: Only reuse existing IRTE allocation for
 Multi-MSI
Content-Language: en-US
To:     <decui@microsoft.com>, <wei.liu@kernel.org>, <kys@microsoft.com>,
        <haiyangz@microsoft.com>, <sthemmin@microsoft.com>,
        <lpieralisi@kernel.org>, <bhelgaas@google.com>,
        <linux-hyperv@vger.kernel.org>, <linux-pci@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <mikelley@microsoft.com>,
        <robh@kernel.org>, <kw@linux.com>, <helgaas@kernel.org>,
        <alex.williamson@redhat.com>, <boqun.feng@gmail.com>,
        <Boqun.Feng@microsoft.com>
CC:     Carl Vanderlip <quic_carlv@quicinc.com>
References: <20220804025104.15673-1-decui@microsoft.com>
From:   Jeffrey Hugo <quic_jhugo@quicinc.com>
In-Reply-To: <20220804025104.15673-1-decui@microsoft.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 8/3/2022 8:51 PM, Dexuan Cui wrote:
> Jeffrey's 4 recent patches added Multi-MSI support to the pci-hyperv driver.
> Unluckily, one of the patches, i.e., b4b77778ecc5, causes a regression to a
> fio test for the Azure VM SKU Standard L64s v2 (64 AMD vCPUs, 8 NVMe drives):
> 
> when fio runs against all the 8 NVMe drives, it runs fine with a low io-depth
> (e.g., 2 or 4); when fio runs with a high io-depth (e.g., 256), somehow
> queue-29 of each NVMe drive suddenly no longer receives any interrupts, and
> the NVMe core code has to abort the queue after a timeout of 30 seconds, and
> then queue-29 starts to receive interrupts again for several seconds, and
> later queue-29 no longer receives interrupts again, and this pattern repeats:
> 
> [  223.891249] nvme nvme2: I/O 320 QID 29 timeout, aborting
> [  223.896231] nvme nvme0: I/O 320 QID 29 timeout, aborting
> [  223.898340] nvme nvme4: I/O 832 QID 29 timeout, aborting
> [  259.471309] nvme nvme2: I/O 320 QID 29 timeout, aborting
> [  259.476493] nvme nvme0: I/O 321 QID 29 timeout, aborting
> [  259.482967] nvme nvme0: I/O 322 QID 29 timeout, aborting
> 
> Some other symptoms are: the throughput of the NVMe drives drops due to
> commit b4b77778ecc5. When the fio test is running, the kernel prints some
> soft lock-up messages from time to time.
> 
> Commit b4b77778ecc5 itself looks good, and at the moment it's unclear where
> the issue is. While the issue is being investigated, restore the old behavior
> in hv_compose_msi_msg(), i.e., don't reuse the existing IRTE allocation for
> single-MSI and MSI-X. This is a stopgap for the above NVMe issue.
> 
> Fixes: b4b77778ecc5 ("PCI: hv: Reuse existing IRTE allocation in compose_msi_msg()")
> Signed-off-by: Dexuan Cui <decui@microsoft.com>
> Cc: Jeffrey Hugo <quic_jhugo@quicinc.com>
> Cc: Carl Vanderlip <quic_carlv@quicinc.com>
> ---

I'm sorry a regression has been discovered.  Right now, the issue 
doesn't make sense to me.  I'd love to know what you find out.

This stopgap solution appears reasonable to me.

Reviewed-by: Jeffrey Hugo <quic_jhugo@quicinc.com>