From: Bart Van Assche <Bart.VanAssche@wdc.com>
To: "linuxppc-dev@lists.ozlabs.org" <linuxppc-dev@lists.ozlabs.org>,
        "abdhalee@linux.vnet.ibm.com" <abdhalee@linux.vnet.ibm.com>,
        "brking@linux.vnet.ibm.com" <brking@linux.vnet.ibm.com>
CC: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "hch@lst.de" <hch@lst.de>,
        "linux-scsi@vger.kernel.org" <linux-scsi@vger.kernel.org>,
        "sfr@canb.auug.org.au" <sfr@canb.auug.org.au>,
        "sachinp@linux.vnet.ibm.com" <sachinp@linux.vnet.ibm.com>,
        "linux-next@vger.kernel.org" <linux-next@vger.kernel.org>,
        "hare@suse.com" <hare@suse.com>,
        "mpe@ellerman.id.au" <mpe@ellerman.id.au>
Subject: Re: [BUG][bisected 270065e] linux-next fails to boot on powerpc
Thread-Topic: [BUG][bisected 270065e] linux-next fails to boot on powerpc
Thread-Index: AQHTFrEsFAqRDNfhzkCHszoyuzdYEKKHOrsAgABjzoCAARXLgIAB6V2AgAAKWYA=
Date: Fri, 18 Aug 2017 21:41:13 +0000
Message-ID: <1503092473.2622.17.camel@wdc.com>
References: <1502902815.3305.22.camel@abdul.in.ibm.com>
         <1502904072.2421.3.camel@wdc.com>
         <2f686064-3e32-df8d-134f-962b5181da9d@linux.vnet.ibm.com>
         <1502985161.2615.8.camel@wdc.com>
         <71fb9c1b-9f3f-acdc-8bb5-aa1240aea763@linux.vnet.ibm.com>
In-Reply-To: <71fb9c1b-9f3f-acdc-8bb5-aa1240aea763@linux.vnet.ibm.com>
Accept-Language: en-US
Content-Language: en-US
wdcipoutbound: EOP-TRUE
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="utf-8"
Content-ID: <68887D7DDBE50145B91F1308358F04AA@namprd04.prod.outlook.com>
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Aug 2017 21:41:13.9826
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: b61c8803-16f3-4c35-9b17-6f65f441df86
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR0401MB1307
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by nfs id v7ILgBYb030860
Content-Length: 1857
Lines: 32

On Fri, 2017-08-18 at 16:04 -0500, Brian King wrote:
> I think I have an understanding what is going on and why Bart's patch is causing problems for ipr.
> I can work around the boot hang in ipr, but ultimately I think we need to figure out a fix
> in scsi / block. I added some tracing and confirmed its not a matter of commands getting stuck
> in ipr. The issue is we are retrying failed commands until we finally run out of time. This is
> what I see:
> 
> 1. sd_revalidate_disk calls scsi_report_opcode
> 2. ipr RAID arrays don't support MAINTENANCE_IN / MI_REPORT_SUPPORTED_OPERATION_CODES
> 3. ipr returns the command with DID_ERROR
> 4. scsi_decide_disposition goes to maybe_retry, increments scmd->retries, and returns NEEDS_RETRY
> 5. scsi_softirq_done calls scsi_queue_insert to requeue the command, which calls scsi_mq_requeue_cmd
> 6. With Bart's change, we then clear RQF_DONTPREP in this path, while prior we did not
> 7. This results in the command getting scmd->retries zeroed out when it gets re-queued,
>    since we go through prep again and we lose our retry counter, resulting in lots and lots of retries.
> 8. Since the default command timeout for an ipr RAID array is 120 seconds, these retries go on for
>    quite a long time...
> 9. Finally, the command has been retried so long we trip over the overall retry timer
>    in scsi_softirq_done and we timeout the command.
> 
> I'll follow up with a patch to ipr to workaround the hang, but I think we need to somehow preserve
> the retry counter in the scsi command, as this will likely cause issues with other drivers. 

Hello Brian,

Thanks for the detailed analysis. This is very helpful. Have you considered
to change the ipr driver such that it terminates REPORT SUPPORTED OPERATION
CODES commands with the appropriate check condition code instead of DID_ERROR?

Thanks,

Bart.